Unipotent Representations and Quantum Induction 



Hongyu He 
Department of Mathematics 
Louisiana State University 
email: hongyu@math.lsu.edu 

Abstract: In this paper, we construct unipotent representations for the real orthogonal 
groups and the metaplectic groups in the sense of Vogan. In particular, our results imply 
that there are irreducible unitary representations attached to each special nilpotent orbit. 



Introduction 



The existence of certain irreducible unitary representations, often called unipotent representations, 
was conjectured by Arthur, Barbasch and Vogan. Precise predictions about the representation- 
theoretic invariants associated with these representations were given in jAr83| . |BV85j . [V089] 
and jAr] . The goal of this article is to verify the existence of unipotent representations for 0{p, q) 
and Mp2ni^) in the sense of Vogan. The main idea is to study the theta correspondence and its 
compositions with respect to the dual pairs {0{p, q), Sp2nO^)) along the line of |Ho89] |LI89| [HeOOj 
[Heuj Heq . Other classical orthogonal and symplectic groups can all be realized in dual pairs that 



are of type I in the sense of Howe [Ho89] . and unipotent representations for these groups can be 
constructed in the same spirit. 



Unipotent representations play a crucial role in the representation theory of real reductive groups 
and in the theory of automorphic forms. Constructing unipotent representations is one core part of 
Vogan's program to classify the unitary duals of real reductive groups ( |V086j ) . For general linear 
groups over Archimedean fields, unipotent representations can be produced by parabolic induction 
and the classification of unitary duals was carried out by Vogan himself in |Vo86] . For complex 
semisimple groups, unipotent representations were constructed and studied by Barbasch- Vogan 
( |BV85) ). Later, Barbasch proved the unitarity of these representations which led to his classifi- 
cation of the unitary duals of complex classical groups ( [B-89]). For other types of real reductive 
groups, Adams, Barbasch and Vogan gave certain descriptions about special unipotent representa- 
tions in Arthur packets ( [ABV] and |Arj ) . The unitarity and construction of these representations 
remain open problems. 



In this paper, we propose a different way of obtaining unipotent representations that will auto- 
matically be unitary. The motivation came from the work of Segal-Shale- Weil on the oscillator 
representations and the work of Howe, Li, Przebinda and many others on theta correspondences. 
In [Heq], we defined the concept of quantum induction as a composition of theta correspondences 
(see |Ho89] ) within a certain range. Moreover we proved that quantum induction, if nonvanishing, 
preserves unitarity. In this paper, we further study quantum induction in the framework of the 
orbit method ( ^VQ86j . [V094] ) . Using quantum induction, we attach to certain real rigid nilpotent 
orbits O a packet of irreducible unitary representations, M{0). Each irreducible representation 
in J\[{0) satisfies the characterizations given in |V089] . 
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Let me illustrate our construction by an example. Let Od be a real nilpotent coadjoint orbit of 
5p3o(M), with D = 
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Notice that Od is not special in the sense of Lusztig ( |Lus] . |CM] ). Let j be a positive integer. 
Define D — j to be the Young diagram obtained by deleting the first j columns from the left. Let 
Pj be the number of + in D — j and qj be the number of — in D — j. Construct the sequence 

[{Po,qo), {Pi,qi), (P2,g2), • • •], 

obtaining 

[(15, 15), (10, 11), (7, 7), (5, 4), (2,2), (1,0)]. 
Let X be a unitary character of 0(1). Define 

Q(30;10,ll;14;5,4;4;l,0)(x) 
=es{0{W, ll),Mpsom)Os{Mpu{R), 0(10, 11))0.(O(5, 4), Mpi4(M)) (1) 
0,(Mp4(M),O(5,4))0,(O(l,O),Mp4(M))(x). 

where 9s is the theta correspondence in the semistable range (see Definitions [T7] and [22]) . As 
proved in |Heq| , (5(30; 10, 11; 14; 5, 4; 4; 1, 0)(x) is a genuine irreducible unitary representation of 
Mp3o (M) if it does not vanish (see Theorem 13.61 and Theorem 13. 7p . One of the main results of this 
paper is the nonvanishing of Q{*){x) (Theorems 14.2] 15.11 16. Sp . In Chapter 5 and 6, we compute 
the infinitesimal character and the associated variety of Q{*){x) based on the results obtained by 
Przebinda ( |PR96j . |PR93p . We further prove that the associated variety of Q{*){x) is indeed the 
closure of the complexified orbit Oc- 



Here is one of the highlights in this paper. 



Theorem 0.1 Let G = Sp2n{^) or G = 0{p,q) with p + q even. Let be a special rigid 
nilpotent adjoint orbit of Gc parametrized by the partition d (see Ch 6.3,7.3 \CM^ ). Let O be 
a real adjoint orbit of G in Od- Then there exists a nonempty set J\f{0) of irreducible unitary 
representations of G such that for every vr € N{0) the associated variety ofn, V{Anmr) = d(C'd)- 
Let d* = (mi > m2 > > ... > . . .) be the transpose of d (considered as a Young diagram). 
Then the infinitesimal character I (ir) only depends on d. 

• IfG = Sp2n{m, A^) = (P(sp™i (C)), p(o(m2, C)), p(sp^3(C)), p(o(m4, C)), . . .); 
. //G = 0(p,<7), J(7r) = (p(o(mi,C)),p(Bp^^(C)),p(o(m3,C)),p(5p^^(C)),...). 
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Here p{g) is the half sum of the positive roots of q. 
Let me make a few remarks here. 

Remark 0.1 1. Assuming to be rigid and special implies that all rrii must be even. Thus 
all TTij in Theorem 10.11 are assumed to be even. The explanation is fairly simple. On the one 
hand, a rigid orbit is defined to be an orbit that is not induced. The classification of rigid 
orbits for classical Lie algebras, due to Kempken and Spaltenstein, can be found in |CM] . 
Roughly, Od is rigid if d* is multiplicity-free. Special nilpotent orbits, on the other hand, are 
defined by Lusztig in |Lusj . For Lie algebras of type C and D, Od is special if any odd number 
in d* occurs with even multiplicities (see |Lus) also Proposition 6.3.7 [CM] ). Therefore, Od 
being rigid and special forces rrii to be even for all i. 

2. As illustrated by the example for Mp3o(M), our construction of N{0) goes beyond special 
rigid orbits. It produces unipotent representations attached to non-special rigid orbits. The 
most well-known examples of these unipotent representations, are the two irreducible con- 
stituents of the oscillator representation, also called the Segal-Shale- Weil representation. The 
representations constructed in this paper can be regarded as derivations of the oscillator 
representation. 

3. Some of the representations in this paper have been known in one way or another. For 
example, minimal representations, the representations attached to minimal orbits, are studied 
by Brylinski-Kostant in a series of papers ( |BKj ) and by Binegar-Zierau for 0{p,q) ( [BZ] ) 
from a different angle. Beyond that, representations attached to a perhaps wider class of 
small rigid orbits have also been studied by Kashiwara-Vergne, Howe, Li, Sahi, Tan, Huang, 
Zhu and others (see |KV] , |Ho84j , |Sahi| , |HT] , |ZH| , |HL] and the references within them) . 
Our approach is a generalization of the latter. 

4. The intrinsic connection between coadjoint orbits and the unitary dual of a Lie group was 
first explored by Kirillov and Kostant. It is now known as the orbit method. One core part 
of the orbit method for real reductive Lie groups is to construct unipotent representations 
attached to rigid nilpotent orbits. Once we know how to attach representations to rigid 
nilpotent orbits, there are various ways to attach unitary representations to induced orbits 
(see |V094] . |V087| ). What we have accomplished in this paper is the construction of some 
unipotent representations attached to special rigid orbits and some other nonspecial rigid 
nilpotent orbits. It is not clear whether our list of AA(0) exhausts all unipotent representations 
attached to these rigid orbits. At least for some special rigid orbits, it does ( |HL| ) . We 
conjecture that J\f{0) is exhaustive for special rigid orbits. 

5. Similar statements hold for 0{p, q) with p + q odd and for Mp2nO^)- See Theorems 15. H 15.21 
Ea Ea Eland Definition M 

6. Every special nilpotent orbit is induced from a special rigid orbit. Admissible unitary repre- 
sentations associated with induced orbits can be constructed by unitary parabolic induction 
(see |BV85| . [VOOlj ). Let be a special nilpotent orbit of 5p2n(C) or 0(n,C). Our results 
imply that there exists an irreducible unitary representation tt of Sp2nO^) or 0{p,q) such 
that ViAnmr) = d{Od). 
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Also included in this paper is a theorem concerning the relationship between quantum induction and 
parabolic induction. Based on a theorem of Kudla-Rallis |KRj and of Lee-Zhu [LZj, we decompose 
a certain parabolically induced representation I" (it) of Mp2n{^) into a direct sum of quantum 
induced representations Q{p,q){n) with p + q = n + 1 and a fixed parity on p (see Definition [25l 
Theorem 14.51 and Theorem 14. lip . Each Q{p, 9)(vr) is either irreducible or vanishes. This is the limit 
case for which quantum induction can be constructed from parabolic induction. Correspondingly, 
there is a decomposition theorem for the wave front set of /"(vr), namely 

=n+l,p—q=a (mod 4) 

On the one hand, WF{I'^{tt)) is computable and consists of a finite number of irreducible com- 
ponents. On the other hand, the wave front set of each (J')(vr) possesses a certain distinctive 
characteristic that can be derived from [PR93j and |PANj . Under certain hypotheses on vr, we 
sort out the occurrence of WF{Q{p,q){Tr)) in WF{I"{Tr)) completely. This provides us with one 
nonvanishing theorem for Q{p, q){7r) and for 9sip, q)i'^)- 

Let me point out one advantage of our construction related to the theory of automorphic forms. 
Theta correspondence originated from the studies of theta series (see |Sij , |WE65| , |We65| , pHo79] ) . 
As pointed out to me by Jian-Shu Li, theta correspondence should map automorphic representa- 
tions to automorphic representations (see |Ho79j . |Ra87] . |Li94j ). Thus representations in M{0) 
should all be automorphic. In the final part of this paper, along the lines of [V086| . we make some 
conjectures regarding the automorphic dual in the sense of Burger-Li-Sarnak [BLSj . 

I wish to acknowledge my gratitude to Monica Nevins, Shu- Yen Pan, Tomasz Przebinda and David 
Vogan for some very helpful e-mail communications and to Bill Graham, Chen-Bo Zhu, Gestur 
Olafsson and Ray Fabec for reading the manuscript and for their comments. I also wish to thank 
the referee for his very valuable suggestions and comments, in particular his suggestions concerning 
Definition [3T] and the proof of Theorem l4.11[ In the midst of revising this paper, my father Decai 
He, passed away. My father had always encouraged and inspired me in my mathematical endeavors. 
I would like to dedicate this paper to him. 
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Chapter 1 

Invariants 



In mathematics, classification problems are often approached by constructing invariants. In this 
chapter, we attach invariants to the equivalence classes of irreducible admissible Hilbert represen- 
tations of a reductive group G. One hopes that these invariants could shed some light on the 
classification and construction of irreducible unitary representations. Our purpose here is not to 
give a historical account of these invariants, but rather, to review some basic facts we need con- 
cerning these invariants. We adopt the standard notations from [KNj and [Wallach] . 

1.1 Notation 

Reductive groups and semisimple groups in this paper are assumed to have at most a finite number 
of components. Let G be a Lie group. We adopt the following notation: 

1. Go — the identity component of G; 

2. g — the real Lie algebra of Gq; 

3. Q* — the dual space of g; 

4. 0c — the complex Lie algebra of Gq; 

5. U{q) — the complex universal enveloping algebra; 

6. Z{q) — the center of C/(g); 

Let G be a real reductive group. We adopt the following notation: 

1. K — a maximal compact subgroup of G; 

2. n(G) — the set of equivalence classes of irreducible (g, A')-modules; 

3. Iiu{G) — the set of equivalence classes of unitarizable {q,K)- modules, or equivalently, the set 
of equivalence classes of irreducible unitary representations of G. 

4. ( , ) — an invariant quadratic form on g such that ( , )|t is positive definite; 

5. p — the orthogonal complement of t with respect to ( , ); 
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6. a — a maximal Abelian Lie subalgebra of p; 

7. A — the connected Abelian group generated infinitesimally by a; 

8. KAK — a Cartan decomposition. 

Let F be a finite dimensional vector space over F. We use V* to denote HomF(V,F). Let G be a 
real reductive group. 

Notation 1 Let (7r,H) he a Hilhert representation ofG. Fix a maximal compact subgroup K. Let 
V(tt) he the space of smooth K-finite vectors in H. To emphasize n, H is often denoted hy Ht^- 

The space l^(vr) is a (g, ii')-modulc. vr is said to be admissible if each K-type occurs in V{tt) with 
finite multiplicity. If, in addition, V{n) is finitely generated as a C/(fl)-module, then V{n) is often 
called a Harish-Chandra module. 

If vr is unitary and irreducible, then the Hilbert norm is unique up to a scalar multiplication. 
So, equivalence classes of irreducible unitary representations are in one-to-one correspondence with 
irreducible unitarizable Harish-Chandra modules. 

All unitary representations in this paper, unless otherwise stated, are taken as suitable 
unitarized Harish-Chandra modules. 

This convention is consistent with the way we construct irreducible unitary representations. First, 
wc will construct a (g, ii')-module (vr, V). Then we will show that V is an irreducible (g, -fC)-module. 
Hence F is a Harish-Chandra module. Finally, we will prove the existence of an invariant inner 
product on V. Then vr is an irreducible unitary representation of G. For simplicity, we use vr to 
denote the group action and the Lie algebra action. We define three involutions in the category of 
Harish-Chandra modules: 

1. vr*, the contragredient representation; 

2. vr*^, the representation vr equipped with the conjugate complex linear structure; 

3. vr'^, the Hermitian dual representation of vr. 
We have (vr*)"^ = vr'*. If vr is unitary, then vr'^ = vr. 

Notation 2 Suppose a,b E M". Write a <h if and only if for every \ < k < n, 



k 



k 



write a -<b if and only if for every 1 < k <n, 



k 



k 



The ordering :^ is a partial ordering. 
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1.2 Infinitesimal Character and Harish-Chandra Homomorphism 



Let 5 be a complex reductive Lie algebra. Let f) be a Cartan subalgebra of g. Let W{q, f)) be the 
Weyl group generated by the root system f)). Let U{1))^^^'^^ be the space of W{q, [))-invariant 
vectors in U{i)). Then the Harish-Chandra homomorphism 

nC : Z{g) ^ 

is an algebra isomorphism (see jKNj . Chapter VIIL 5 or jWallach] ) . Identify U{i)) with the 
symmetric algebra of f}. For each vector A in the complex dual space of f}, define a character XA of 
Z{q) by 

XAix) = K{rLC{x)) = rLC{x){K) {x G Z{q)) 

It is well-known that every character of Z{q) can be obtained this way and that A is unique up to the 
action of W{q,\)). In short, Spec{Z{Q)) = ^* / /W{q,\)). Here the categorical quotient ^* / /W{q,\)) 
coincides with the geometric quotient since W{q, f)) is finite. 

Let G be a connected real reductive Lie group with Lie algebra 3. Let be a maximal com- 
pact subgroup of G. Let (vr, 7i) be an irreducible admissible Hilbert representation of G. Let F(vr) 
be the Harish-Chandra module of (vr,?^), consisting of all the ET-finite vectors in the Hilbert space 
7i. We retain vr for the infinitesimal action of U{q) on the smooth vectors of (7r,7i). Since vr is 
irreducible, V{tt) is an irreducible ([/(g), K)-module. Since G is connected, Z{q) = U{q)^. By 
Schur's lemma, Z{q) acts on F(vr) by a character 

X : Z{q) ^ C. 

Thus there exists a A such that Z[q) acts on V^('/r) by Xk- For simplicity, we call A an infinitesimal 
character of vr. 

Let G be a real reductive group with a finite number of components. Let f) be a complex Cartan 
subalgebra in gc- Then [/(g)*^" = Z{q). Let U{q)^ be the G-invariant vectors in U{q). Consider 
the adjoint action of G/Gq on Z{q). Each element in G/Gq acts on Z{q) as an algebra automor- 
phism. Furthermore, U{g)'^ is precisely the subalgebra of Z{q) invariant under the action of G/Gq. 
By the Harish-Chandra homomorphism, G/Gq acts on 

as algebra automorphisms. This action induces an action of G/Gq on 

Spec(C/(l))^(9c,f))), 

Since G/Gq is finite, by invariant theory 

Spec(?^C(C/(0)^)) 
is precisely in one-to-one correspondence with the G/Go-orbits in 

Spec(?7(f))^(9^''')). 

Let TT be an irreducible (g, i<')-module. By Schur's lemma, U{q)'^ must act by a character ^. 
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Definition 1 Let ^ he a Cartan subalgebra of qq. We say that A E {)* is an infinitesimal character 
of-n- ifXA\u(g)G = 6 

A is unique up to tlie action of G/Gq and Ty(gc,f)). For semisimple Lie group G, the action of 
G/Gq on f)*//VF(0C) [)) is not difficult to understand. Let l be the action of G/Gq on f)*//VF(0c5 
induced from the action of G/Gq on q* / /Gq. Then i induces an action of G/Gq on a closed Weyl 
chamber, consequently on the positive root system. Hence l{G/Gq) can be regarded as automor- 
phisms of the Dynkin diagram of gc. The automorphisms of the Dynkin diagram are not difficult 
to classify. 

The infinitesimal character is one of the main tools used in the literature to study irreducible 
admissible representations. In fact, there is only a finite number of infinitesimal equivalence classes 
of irreducible representations with a fixed infinitesimal character. 

Notation 3 Let Yi\[G) he the set of infinitesimal equivalence classes of irreducihle admissible 
Hilhert representations of G with infinitesimal character A. 

1.3 Leading Exponents of Irreducible Representations 

Let G be a real reductive Lie group. Fix a maximal compact subgroup K. Unless otherwise 
stated, matrix coefficients in this paper are assumed to be JC-finite. Fix a nondegenerate real 
invariant bilinear form ( , ) on g and a maximal Abelian Lie subalgebra a of p as in (1.1). Let 
r be the real dimension of a. We call r the real rank of G. Let a) be the restricted root system. 

Fix a positive root system I]+(3,a). Let p{G) be the half sum of all positive roots in S(g,o). 
Let M be the centralizer of o in K. Let W{G, a) be the normalizer of a in ii' modulo M. We 
call W{G, o) the real Weyl group. Let W{q, a) be the Weyl group generated by the root system. 
Clearly, W{Q,a) C W{G,a). 

For S'p2n(IR), CI is isomorphic to R". The real Weyl group W{Sp2n(^)-,c) is generated by per- 
mutations and sign changes on n variables. For 0{p, q), o is isomorphic to M™™'!?-'?!. The real Weyl 
group W{0{p, q), a) is also generated by permutations and sign changes on mm{p, q} variables. See 
1.1.1 and 1.3.2. 

Attached to each irreducible admissible representation tt of a connected G, is a finite number of 
vectors in a^, called leading exponents. Leading exponents are the main data used to produce the 
Langlands classification (see |LA| . |KNj . [Wallachj ) . For connected G, leading exponents depend 
on the choice of S^(g, a). For G with finite number of connected components, we define the lead- 
ing exponents of vr € n(G) to be the leading exponents of the irreducible subrepresentations of it\go- 

Leading exponents are closely related to the infinitesimal character. For an irreducible finite di- 
mensional representation vr of a connected real reductive group G with real rank equal to complex 
rank, leading exponent is just the highest weight of vr with respect to a maximally split Cartan 
subgroup. In this situation, finite dimensional representation theory says that the highest weight 
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V is related to the infinitesimal character A by the following equation 

V + p = wA 

for some w in W{G, f)). A similar statement holds for leading exponents for any irreducible admis- 
sible representation of a real reductive group. By Theorem 8.33 from [KNj . we have 

Theorem 1.1 Let b be a Cartan subalgebra of m. Take f) = a© b. Suppose vr is an irreducible 
admissible representation of a real reductive group G. Let v be a leading exponent of n. Then there 
exists an infinitesimal character A of tt in [)^ such that 

v + p{G)=A\a. 

Notice that for non-split groups, v is in and A is in (a © b)J. By Theorem II. H for any vr G IIa, 
the set of leading exponents is finite. 

Notation 4 For v £ a^, the complex dual of a, we denote the real part of v by 3f?(f). 

Leading exponents are extracted from a certain asymptotic expansion of the matrix coefficients at 
oo. Therefore, they control the growth of matrix coefficients. We cite the following estimate from 

Theorem 1.2 Let vr be an irreducible admissible representation of a real reductive group G. Let 
vq be in the real dual of a. Let o"*" be the closed Weyl Chamber associated with £+(9,0) (see Prop. 
5.14 \KN^ ). Let be the closed Weyl Chamber in A obtained by exponentiating a"*". Let a{g) be 
the middle term of the KA^K decomposition of g. If every leading exponent v of n satisfies 

{vo - ^{v)){H) > (VifGo+), (1.1) 

then there is an integer q > such that each K -finite matrix coefficient of vr is dominated by a 
multiple of 

exp(?;o(loga(c/)))(l + (log a(g), log a(c/)) 2 

The converse also holds. 

When G = 0{p,p), due to the action of G/Gq, A+ and a+ can be made smaller (see 1.3.1). Theo- 
rem 11.21 remains valid for this generalized Weyl chamber A~^. 

Theorem 11.21 provides a uniform bound for matrix coefficients for all vr € IIa{G). Let vr be a 
unitary representation in Il/^{G). If A is "small", we can easily find a vq to dominate the set 

{wA\a-p{G) \weW{G,t)c)}- 

By Theorem 11.21 we gain fairly good control over the growth of the matrix coefficients for all 
vr G IIa. If A is large, then the convex cone spanned by the set 

{wA\a-p{G)\wGW{G,\:,c)} 
is widely spread. The uniform bound from Theorem 11.21 does not yield any useful information. 

For unitary representations in n(G), the matrix coefficients are all bounded by constant functions. 
Theorem 11.21 then implies 
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Corollary 1.1 Let G be a real reductive group. If tt n„(G), then every leading exponent v of it 
satisfies 

^{v){H) < (y H e a+). 

In this paper, we will mostly deal with small A. To give a broader picture of the importance of 
UuiG) with small infinitesimal character, we recall one definition from [VOOOj and ^SVj . 

Definition 2 A unitary representation it £ nA(G) is called "unitarily small "if A is in the convex 
hull spanned by 

{w{p{Qc)) I w £ W{0,f))}. 

Salamanca- Vogan conjectured that any irreducible unitary representation of G not small may be 
constructed by parabolic or cohomological induction from an irreducible unitarily small represen- 
tation of a reductive subgroup of G ( [SV] , |VO00) ). 

It is precisely for "unitarily small "representations that Theorem 11.21 will produce useful infor- 
mation about matrix coefficients beyond what is stated in the corollary. One goal of this paper is 
to construct and study some unitarily small representations far away from the tempered unitary 
representations. 

1.3.1 Example I: The groups Mj92n(IR) and Sp2nO^) 
Let G be either Mp2n(K) or Sp2n(^)- Fix 

a = {diag(ai,a2, ...an, -ai, -02, • • • - an)}- 

Then the Weyl group W{G,a) is generated by permutations and sign changes on {ai}f=i. Fix 
positive roots S"*" = {cj ±ej \ i < j; i,j € [1, n]}. Then 

p{G) = (n,n-l,...,l) 

and 

0+ = {diag(ai,a2,...,an,-ai,-a2,...,-a„) | oi > 02 > . . . > a„ > 0}. 

Clearly, 

A+ = {dmg{Ai,A2,...An,A^\A2\...A;;^) | vli > ^2 > ■ ■ • > > !}• 
Finally, the Inequality ( II. ip is equivalent to 

(see Notation [2] and 4). 

1.3.2 Example II: The Groups 0{p,q) 

Let G = 0{p, q) with q > p. Here 0{p, q) is the isometry group of 

p Q-p 

(a;, y) = "^{xiyp+i + Xp+iVi) + ^ X2p+iy2p+i- 

i=i 3=1 
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Fix 

a = {diag(ai, 02, . . . Op, -ai, -02, . . . - Op, 0, 0, . . . 0) | € M}. 

Then the Weyl group W(G,a) is again generated by permutations and sign changes on {cj}^^^. 
Notice for p = q, W{G, a) is bigger than W{g, 0). 

Fix restricted positive roots 

E+ = {a ±ej \ i < £ [l,p\} U {e, | i € [l,p]} ii p < q 

and 

S+ = {ei ±ej \ i < € if p = g. 

Then 



Fix a (generahzed) Weyl chamber 

a+ = {diag(ai, 02, . . . Op, -ai, -02, . . . - Op, 0, 0, . . . 0) | ai > 02 > . . . a„ > 0}. 

Then 

= {dmg{Ai,A2,...Ap,A^\A^\...,A-^) \Ai>A2>...Ap>l}. 

The Inequahty ( ll-lh is again equivalent to vq >z R{v). Notice that for 0{p,p), due to the action of 
0{p,p)/ SOq{p,p), A~^ is half of the Weyl Chamber determined by S"^. 



1.4 Global Characters 

The global character is also known as the Harish-Chandra character. For each irreducible admissible 
Hilbert representation vr of G and a compactly supported smooth function / on G, define G(7r)(/) 
to be the trace of the operator 

vr(/) = J fi9)n{g)dg. 

A theorem of Harish-Chandra states that is a distribution on G which can be identified with 
a locally integrable function (still denoted by 0-,^) and 0,^ is real analytic on the set of regular 
semisimple elements of G (see [HC| ) . 

For vr unitary, Milicic defined a notion of the rate of growth of ©^r which we will not recall here. 
We will state one theorem relating 7 to the matrix coefficients of vr (see [Ml]). 

Theorem 1.3 (Thm. 1 [MI] ) Let it £ Ilu{G) . Let'E{g) be Harish-Chandra 's E function. Then 
the following are equivalent: 

1. ©7r has the rate of growth 7 G M; 

2. every K -finite matrix coefficient of tt is bounded by 

CH^-^(5)(1 + II log(a(<7))||)^ {C,s> 0,a{g) G 
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3. every leading exponent v of tt satisfies 

^{v){H) < {j-l)p{G){H) (yHea+). 



So if 7 = then vr is tempered and 7 = 1 if vr is trivial. For G = Mp2n(^) or G = 0{p, q), the last 
statement is equivalent to 

^{v)^{j-l)p{G). 

1.5 Associated Variety, Asymptotic Cycle and Wave Front Set 

Recall that U{q) has a natural filtration 

C = Uoie) e Uiis) C U2iQ) C Usid) C . . . C Unid) c . . . . 

1.5.1 Annihilator and Associated Variety 

Let vr be an admissible irreducible representation of a reductive group G. Let T^(vr) be the Harish- 
Chandra module of vr. Then ^^(vr) is a U{q) module. Consider the annihilator of V{tt), 

Ann{V{TT)) = {De U{g) \ tt{D)V{tt) = 0}. 

Ann(y (tt)) is an ideal of U (g) and inherits a filtration from the standard filtration of U{q). It follows 
that the induced graded algebra gr{Ann{V{-K))) is an ideal of gr{U{g)) = S{g) and necessarily 
commutative. 

Notation 5 Let V{Ann tt) be the associated variety of gr{Ann(V{TT))). 

Theorem 1.4 (Borho-Brylinski [BB], Joseph [Jo]) Suppose G is a connected semisimple group 
and vr is an irreducible representation of G. Let q* be the dual space of Qc- Then V{Ann vr) is the 
closure of a single nilpotent orbit in . 

For G having a finite number of components, V{Ann vr) will be the closure of a finite number of 
nilpotent coadjoint orbits of equal dimension. In fact, G/Gq acts on the set of nilpotent coadjoint 
orbits. The associated variety V{Ann vr) will be the closure of the union of one nilpotent orbit with 
its translations under Ad{G/Go). For a detailed account of associated varieties of Harish-Chandra 
modules, see [V089| . 

1.5.2 Asymptotic Cycle and Wave Front Set 

Let G be a semisimple Lie group with finitely many components. Let us consider the global 
character Q^^. One can lift G,r to an invariant distribution D on g. There exists a Taylor expansion 
of D near 0, 

00 

D{f{tx)) - J2 *^A(/(x)). 

i=—r 

In |BV80] ■ Barbasch-Vogan proved that the Fourier transform Di is supported on the nilcone of 
0*. 
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Notation 6 Let supp{AS{'n)) he the closure of the union of supports of Di. We call supp{AS{tt)) 
the support of the asymptotic cycle of it. 

Theorem 1.5 (Barbasch-Vogan) Suppose G is a connected semisimple Lie group. Then the real 
dimension of supp{AS (it)) is equal to the complex dimension ofV{Ann vr). supp{AS{tt)) is a union 
of nilpotent orbits contained in g n V{Ann vr). Let r = ^lE^L^i^ _ Then Z)_,. is the lowest nonzero 
term of the Taylor expansion of D and supp{D-r) is of maximal dimension in supp{AS (tt)) . 

This theorem holds for G with a finite number of components. See |BV80j . 

Another notion similar to ^^^^(^^(Tr)) is the wave front set of tt defined by Howe ( |Ho81j ). 
Originally, WF{7r) was defined as a closed subset of the cotangent bundle T*G. Because of the 
G-action, WF{Tr) can be regarded as a closed subset of g*. For G semisimple and tt irreducible, 
WF{Tr) is in the nilpotent cone of g*. In [Ho81] . Howe studied the behavior of wave front sets 
WF{tt) under restrictions to certain subgroups. 

Suppose that tt is irreducible and G is semisimple. Rossmann proved that WF{tt) and supp{AS{tt)) 
are identical (see Theorem C, |Roj ) . In what follows, we will not distinguish between WF{tt) and 
supp{AS{tt)) as long as vr is irreducible. There are two basic facts the reader should keep in mind. 
The first fact is that WF(tt) lies in the real space g*. The second fact is that the algebraic closure 
of WF{tt) is exactly the associated variety V{Ann tt) C Qq. 

Notation 7 From now on, we will identify g* with g using a fixed invariant bilinear form on g. 
We will also identify g^ with gc as complexifications of q* and g. 
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Chapter 2 

Nilpotent Orbits 



An element x in a Lie algebra g is called nilpotent if ad{x) is nilpotent. Let G be a Lie group 
with Lie algebra g. If x is nilpotent and g (z G, then Ad{g)x is also nilpotent. It follows that 
under the adjoint action of G, nilpotent elements are grouped into G-orbits. Each G-orbit is called 
a nilpotent adjoint orbit. For semisimple Lie groups, there are finitely many nilpotent orbits and 
the classification of nilpotent adjoint orbits is completely known. We cite Collingwood-McGovern's 
book |CM| as the main reference for this chapter. Unlike in [CM] , nilpotent orbits in this paper 
depend on the Lie groups, not just on the Lie algebra. In particular, for the orthogonal groups, 
a nilpotent orbit may not be connected. For our convenience, we state the Springer-Steinberg 
theorem slightly differently than in [CM] . The reason for doing this is given in Theorem 12. 6i The 
main results are Theorems 12. 3| 12.61 12.71 



2.1 Young Diagrams and Complex Nilpotent Orbits 

A sequence of positive integers 

d = {di > d2 > ■ ■ ■ > dr-i > dr > = dr+i) 
is said to be a partition of n if n = dj. Write ||d|| = n. 

Notation 8 In this paper, a partition of n will he represented by a Young diagram of n boxes, 
arranged as follows: 

1 

□ 
□ 

with the i — th row of length dj. The transpose of d is denoted by d*. 

In our notation, the Young diagram above can be written as d = (7,3,1,1) and its transpose 
d* = (4,2,2,1,1,1,1). 

Definition 3 If dj is odd for every j < r, we say d is very odd. If dj is even for every j < r, we 
say d is very even. If the nonzero dj are all distinct, we say d is multiplicity free. A row of even 
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length is called an even row. A row of odd length is called an odd row. 

2.1.1 Complex Nilpotent Orbits of Sp2n{C) 

Definition 4 A Young diagram d is said to be a symplectic Young diagram of size 2n if odd rows 
occur with even multiplicity and ||d|| = 2n. We use yD-{2n) to denote the set of symplectic Young 
diagrams of size 2n. 

A symplectic group is the linear group that preserves a nondegenerate skew-symmetric form. 

Theorem 2.1 Nilpotent adjoint orbits of Sp2n{^) in one to one correspondence with yD-{2n). 

We denote the nilpotent adjoint orbit corresponding to d by Od{—). The subscript — is used to 
indicate that the group G is defined with respect to a s/^eio-symmetric form. If Od(— ) is known 
to be symplectic, wc will simply write O^- For our convenience, wc denote the symplectic group 
'S'p2n(C) by G{Od{—)) or G{Od). The group Sp2n{'C) is thus attached to the orbit implicitly. 

2.1.2 Complex Nilpotent Orbits of 0{n, C) 

Definition 5 A Young diagram d is said to be an orthogonal Young diagram of size n if even rows 
occur with even multiplicity and ||d|| = n. We use yD^{n) to denote the set of all orthogonal 
Young diagrams of size n. 

An orthogonal group is a linear group preserving a nondegenerate symmetric form. 

Theorem 2.2 Nilpotent adjoint orbits ofO{n,C) are in one to one correspondence with yD^{n). 

We denote the nilpotent adjoint orbit corresponding to d by Od(+)- The script + is used to indicate 
that the group G preserves a symmetric form. If the orbit Od{+) is known to be orthogonal, we 
will simply denote it by Od- In this context, the orthogonal group 0(n, C) is denoted by G{Od{+)) 
or G{Od). 

Depending on the context, Od can refer to either Od(+) or C)d{—). 

2.1.3 Operator -1 

Definition 6 Let d be a Young diagram of size n. Define d — 1 to be the new Young diagram 
obtained by deleting the first column of d. Define d — i to be the new Young diagram obtained by 
deleting the first i columns of d. 

Consider d = [3^, 2^, 1^]. The Young diagrams d, d — 1 and d — 2 are listed as follows 
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□ 
□ 



Notice that d is a symplectic Young diagram, d — 1 is an orthogonal Young diagram, and d — 2 is 
a symplectic Young diagram. 

Theorem 2.3 If d is a symplectic Young diagram, then d — 1 is an orthogonal Young diagram. 
If d is an orthogonal Young diagram, then d — 1 is a symplectic Young diagram. 

2.2 Signed Young Diagrams and Real Nilpotent Orbits 

Type I classical groups are subgroups of the general linear groups that preserve certain sesquilinear 
forms (see [LI89J for the definition). Let G be a real classical group of type I. Real nilpotent orbits 
of G are simply nilpotent G-orbits in the real Lie algebra q. For type I classical groups, the real 
nilpotent orbits for G are parameterized by equivalence classes of signed Young diagrams. 

Definition 7 ( Ch. 9.3 [CM) ) Signed Young diagrams are Young diagrams with + or — labeling 
the boxes in such a way that signs alternate across rows. Two signed Young diagrams are considered 
to be equivalent if one signed diagram can be obtained from the other by interchanging the rows of 
same lengths. 

Let D be a signed Young diagram. We will use to denote the number of positive boxes in D 
and D~ to denote the number of negative boxes in D. We call {D^ , D~) the signature of D. 

In this chapter, we are only interested in the nilpotent orbits of Sp2nO^) and 0{p,q). 

Notation 9 For the sake of our discussion, we fix a matrix realization for each G. So the group 
0{p,q) and Sp2nO^) in this paper, will come with a fixed sesquilinear form. 

The correspondence between signed Young diagrams and real nilpotent orbits depends on the 
sesquilinear forms. I would like to thank the referee for pointing this to me. 

2.2.1 Real Orbits of Sp2n(M) 

Theorem 2.4 (Springer and Steinberg) Nilpotent adjoint orbits of Sp2ni^) o,re parametrized 
by the equivalence classes of signed Young diagrams with the following properties: 

• the signature o/D is {n,n); 

• for every s, rows of length 2s + 1 must occur with even multiplicity and must have their 
leftmost boxes labeled 

from the highest row to the lowest row. 
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Clearly, the signed Young diagrams that satisfy our second condition must have signature {n,n). 
So, the first condition is redundant. In the second condition, the choice of the sign pattern of rows 
of odd lengths is artificial. In fact, it suffices that there are same number of boxes labeled + and — 
in the rows of length 2s + 1. Nevertheless, I have included this in Theorem 12.41 for two purposes. 
The first is to maximize the analogy with Theorem 12. 5[ The second is to make the signed young 
diagrams easy to manipulate. 

We denote the set of signed Young diagrams in Theorem 12.41 by yD-(n,n). The following signed 
Young diagrams are in yD^{n,n): 



l-l+l-l+l-l+l-l 


l-l+l-l+l-l+l 


l+l-l+l-l+l-l+l 


l-l+l-l+l-l+l 


l+l-l+l-l 


1 - l+l - l+l - 1 


l+l-l 


l+l - l+l - l+l 


□ 


1 - l+l - l+l - 1 


s 


l+l - l+l - l+l 



Notice that Afp2n(IR)-adjoint orbits coincide with Sp2n{^)- adjoint orbits. 

Notation 10 We denote the nilpotent orbit corresponding to D by 0-£){—) or simply Od ^/D is 
specified to be symplectic. We denote the group Mp2n{^) by G{Od). 

The reader should notice that G(C'd) is a complex group of which G^O-d) is a real form. 

Definition 8 Define 

^ (xesp2„(M)). 
Notice that 

(10 \(0 In\( I \ In\ 

\0 -I J \ -In J \ -I J \-In J 

and changing the symplectic form ( , ) to — ( , ) results in the same group Sp2ni^)- Therefore r 
defines an involution on 5p2„(M) and on Sp2nO^)- By sorting out the signs in the proof of Lemma 
9.3.1 in [CM] , we obtain 

Lemma 2.2.1 1. r defines an involution on the real Lie algebra 5p2ni^)- 

2. T induces an involution on the set of nilpotent orbits, namely 

Here t(D) is the signed Young diagram obtained by switching the signs (+ —) for the even 
rows of D . 

3. The involution t defines a diffeomorphism from O-q onto Ct(d) • 
The proof is left to the reader. 

Notice that the odd rows of r(D) remain the same as those of D. For example, r: 
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I-I+I-I+I-I+I- 
I+I-I+I-I+I-I+ 



l-l+l-l+l-l+l-l 
l+l-l+l-l+l-l+l 



+E 



EH 

□ 

S 



□ 

m 



One can further explore the involution r on the Lie group level. 



Definition 9 1. Define r on Sp2ni^) by 




Clearly r defines an involution on Sp2nO^)- t is a topological homeomorphism and an isom- 
etry with respect to a certain left invariant Riemannian metric. 

2. Lift T to an involution on Mp2n{^)- The lift r exists and is unique. By abusing notation, 
denote this involution by r. 

3. Let vr G I{{Mp2n{^))- Define a new representation (Tr'^,?^,^) by 



Lemma 2.2.2 vr — > vr"^ defines an involution on Il{Mp2n{^)) o-n-d on n„(Mp2n(l^))- 
2.2.2 Real Orbits of 0{p, q) 

Theorem 2.5 (Springer & Steinberg) Nilpotent adjoint orbits of 0{p, q) are parametrized by 
equivalence classes of signed Young diagrams with the following properties 

• the signature o/D is {p,q); 

• for every s, rows of even length 2s must occur with even multiplicity and must have their 
leftmost boxes labeled 



from the highest row to the lowest row. 

Again, the choice of sign patterns for rows of even length is artificial. We denote the set of signed 
Young diagrams in Theorem 12.51 by 3^D+ (p, q) . The following signed Young diagrams are in 




yD+{7,9y. 



l+l-l+l-l+l-l 
l-l+l-l+l-l+l 

EEH 
□ 



l-l+l-l+l- 
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Notation 11 We use Od to denote the nilpotent orbit corresponding to D € yD^(p,q). We use 
G(Od) to denote the group 0{p,q). Any D in this paper is interpreted as a signed Young diagram 
in a previously chosen set yD^ or 3^D+. The orbit Od refers to either Od(+) or Od(— ) with the 
understanding that + or — is implicitly known once D is given. 

Of course, if D is specified to be in 3^1)+ or yD^, the notation Od causes no confusion. 

2.3 Nilpotent Orbits of Class U 

Notation 12 Let Od be a real nilpotent orbit. We use d to denote the Young diagram obtained 
from D by removing the signs. 

Every real nilpotent orbit O induces a complex nilpotent orbit Oc by considering the complex 
group Gad acting on O in g. This map is simply 



for orthogonal groups and symplectic groups. 

Definition 10 Let Y) be a signed Young diagram, 
obtained from D by deleting the first column. 



Define D — 1 to be the signed Young diagram 



Theorem 2.6 —1 defines an operation from yD^{p,q) to the disjoint union 

^n<mm{p,q)yD-{n,n) 

— 1 also defines an operation from yD-{n,n) to the disjoint union 

Umax(p,g)<n3^^+(P,9) 

By Theorem 12.41 and Theorem 12. 5| —1 induces an orbital correspondence from real nilpotent orbits 
of symplectic groups to real nilpotent orbits of orthogonal groups, and conversely. 

Definition 11 We say that Oj) (or O^) is pre-rigid if is multiplicity free. 

This amounts to saying that every integer between 1 and di appears in the partition d. In other 
words, the partition d is of the following form 

([di]*,[di-l]*,[di-2]*,...,[l]*) 

with each multiplicity greater than zero. 

Definition 12 (Nilpotent orbits of Class U) 

1. lA consists of a class of real nilpotent orbits Od- For technical reasons, we exclude from lA 
those D whose last 2 columns are of the same length with at least two rows and are of the 
following forms 
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EE 
EE 
EE 
EE 
EE 



2. Let D G yD^(n,n) and d* = (mi > m2 > • • • > rud^). The nilpotent orbit Od of Mp2n 
is said to be in U{Mp2nO^)) if d-* satisfies the following three conditions: 

(a) For every i, m2i > m2i+i and m2i+i > rn2i+2; 

(b) (1); 

(c) d* is either very even or very odd. 

3. Let D G yD+{p,q) and 

d* = (mi > m2 > • • • > TTidi). 
The nilpotent orbit Od is said to be inU{0{p,q)) if satisfies the following three conditions: 

(a) For every i, m2i > m2i+i and m2i+i > m2i+2; 

(b) (1); 

(c) d* is either very even or very odd. 

Here are two D, one in Z^(Mp3o(M)), the other in U{0{lb, 22)). 



I-I+I-I+I-I+I 


l+l - l+l - l+l-l 


1 - l+l - l+l - 1 


I-I+I-I+I-I+I 


l+l-l+l-l+l 


1 - l+l - l+l - 1 


1- l+l -l+l 


1 - l+l - l+l - 1 


l+l - l+l - 1 


1 - l+l - l+l - 1 


l+l-l 


1 - l+l - l+l - 1 


l+l-l 


1 - l+l - 1 


□ 


□ 




□ 



The condition (a) is weaker than saying d* is multiplicity free. The condition (c) amounts to saying 
that all multiplicities of d except perhaps the first are even. Recall 



Lemma 2.3.1 (Thm. 7.3.5 and Prop. 6.3.7 [CM] ) If is rigid, then is pre-rigid. If 
Od is special and rigid, then d* must be either very even or very odd and must be multiplicity free. 



Corollary 2.1 Special rigid orbits ofO{p,q) and Sp2n{ 
By Theorem 12.61 we have 



are contained in U. 



Corollary 2.2 // Od is in U{Mp), then Od-i is in U{0). If Od is in U{0), then Od-i is in 
U{Mp). 
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2.4 Induced Orbits 

2.4.1 Complex Induced Orbits 

Let be either o(n, C) or 5p2„(C). Let p = [ n be a parabolic subalgebra. Let Oi be a nilpotent 
orbit in [. 

Definition 13 Define Ind^Oj to be the nilpotent G-orbit that intersects Ot + n in an open dense 
set. 

Our definition differs slightly from the definition in Ch. 7.1 of |CMj as we require Ind^O( be a 
G-orbit and G may not be connected. In any case, Og is the G-orbit of the "generic" elements in 
0{ + n. In addition, Ind^O( only depends on [, not on the choice of p (see Ch. 7.1 [CM] ). 

Lemma 2.4.1 (Proposition 7.1.4 |CM] ) Let li and [2 be two Levi subalgebras of q and li C 
Then 

IndS(Ind|jO(J=IndSO[,. 
Thus orbital induction is "associative". 
If = sp2„(C), then any Levi subalgebra [ is of the form 

t = sp2no(C) X 0Kni,C) X ... X Ql{nr,C) 

where no,ni,... ,nr are nonnegative integers summing to n. Similarly, if g = o(m,C), then any 
Levi subalgebra [ is of the form 

[ = o(mo, C) X g[(mi, C) x . . . x g[(ms, C) 

where mo, nii, . . . are nonnegative integers and ttiq + 2mi -1- . . . + 2ms = 

Recall that nilpotent orbits of SL{n, C) are parametrized by partitions of n in terms of the Jordan 
form. 

Lemma 2.4.2 Let n = ni + 77-2. Let q = sl(n,C). Let I be the block- diagonal matrices of size 
(^1,112) in Q. Let Os and Ot be nilpotent orbits in sl{ni,C) and 5l{n2,C) respectively. Then 

IndfOs xOt = Od 

with 

dj = sj+tj (Vj). 
Definition 14 We call d the merging of s and t. 

For G = 0{n,C) or G = 5p2n(C), computation of induced orbits is slightly more complicated 
than simply a "merging". One needs the concept of "collapse" (see Ch. 6 |CM| ). For the sake 
of simplicity, we will not introduce the concept of "collapse". We only state one special result 
concerning S'p2n(C). The general result can be found in [CM] ( Lemma 6.3.3 ) and is due to 
Gerstenhaber. 
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Lemma 2.4.3 Let G = 5'p2n(C) and L = GL(n — m,C) x S'p2m(C). Let Og be a nilpotent orbit 
of Sp2mi'C) such that the number of rows in s is less than or equal to n — m. Then 



lndf{0} X = Od 

with 

dj = Sj + 2 (V 1 < J < n - m). 

So under the assumption that the number of the rows in s is less or equal to n — m, d is the merging 
of s with two copies of f". d remains symplectic since s is symplectic. 

2.4.2 Induced Real Orbits 

Let G be a reductive Lie group and L be a Levi subgroup. Let P = LN be a parabolic subgroup 
of G. Let Oi be a nilpotent orbit in (. 

Definition 15 Define IndfOi to be the union of the nilpotent G-orbits Og ^ that contain an open 
subset of 0{ + n. 

Implicit in the notation is the fact that Ind^ does not depend on the choice of n. The proof of 
this fact is essentially the same as the proof for the complex orbits (see Theorem 7.1.3 of |CMj ). 
Ind^C[ for a classical group of type I may no longer be a single nilpotent orbit. Nevertheless, it is 
contained in a single complex nilpotent orbit. 

Lemma 2.4.4 We have the following commutative diagram: 



„ complex orbit 

0[ > 



Ind 



Ind (2.1) 



Let Os be a nilpotent orbit in sp2m(I^)- By the lemma above, 

must be contained in the complex induced orbit 

Ind^P2„(C) ,„,{0}xOs. 

Under the assumption in Lemma 12.4.31 this complex induced orbit is parametrized by d with 

~ '^i + ^ (V 1 < j < n — m) . 

So 

is contained in the intersection of with sp2n(^^)- We have 
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Theorem 2.7 Let S G yT)-{m^m) and s = (si > 52 ^ • • • ^ -^r > 0). Suppose that n — m > r. 
Theft 

Each D(j) G yV_{n , n) is uniquely defined as follows: 

1. merge one column of length n — m to S from the left; 

2. merge one column of length n — m to S from the right; 

3. extend the signs of S for rows of even lengths; 

4- fill in the n — m — r rows of length 2 in D^-'^ with j I " l~H s and n — m — r — j l~H~ I s; 

5. the signs o/D^-'^ for rows of odd length are uniquely dictated by the rules we set for yD^{n, n), 
thus not subject to change. 

We give an example first and a sketch of tlie proof will follow. 

Example Let t = be the partition oi n — m which corresponds to the zero orbit for 

GL(n — m, M). Then we can write the conclusion of Theorem 12.71 as 

Suppose S is the signed partition in the middle of the diagram below and n — m = 7. 



□ 


l-l+l-l+l-l 


□ 


□ 


l+l-l+l-l+l 


□ 


□ 


l-l+l-l+l 


□ 


□ 


l-l+l 


□ 


□ 


l-l+l 


□ 


□ 




□ 


□ 




□ 



The procedure explained in Theorem 12.71 will result in the following three signed symplectic Young 
diagrams: 



-l+l-l+l-l+l-l 


l-l+l-l+l-l+l-l 


l-l+l-l+l-l+l- 


+1- l+l-l+l-l+l 


l+l -l+l -l+l -l+l 


I+I-I+I-I+I-I+ 


+I-I+I-I+I-I 


l+l-l+l-l+l-l 


l+l-l+l-l+l-l 


+1 - l+l - 1 


l+l - l+l - 1 


l+l -l+l-l 


+1 - l+l - 1 


l+l - l+l - 1 


l+l -l+l-l 


+l-l 


l+l-l 


l-l+l 


+1-1 


l-l+l 


l-l+l 



These three signed Young diagrams are dW, D(i) and D^^). 
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Sketch of the proof: (1), (2) are evident by the commutative diagram 12.11 and (5) follows from 
our convention (see Theorem I2.4|) . Let 

n—m—r 

d = (si + 2, S2 + 2, . . . , + 2, 2^"^). 
An odd number 2/ + 1 must occur even times. We group 2/ + I's into pairs. 

We adopt the notation from Ch 5. 2 of [CM] and the main reference is Ch 5.2 and 9.3 of [CM]. A 
signed Young diagram can be expressed as 

af e e . . . 

with of symbolizing a row of length aj beginning with the sign it. Notice that odd number 21 + 1 
always comes in pairs as {{21 + 1)"^, {21 + 1)~) according to our convention. 

Fix a symplectic space (M^",il) and a nondegenerate symplectic subspace R^"^. Decompose M^"" 
into 

such that Vq Iq' is the orthogonal complement of M'^™' with respect to Q. and Vq and Vq are both 
isotropic subspaces. Then Qly^^yj is nondegenerate. Let 

n = {u G sp2„(M) I uVq = 0, n(M2'" © Vq) C Vq}. 

It is easy to see that for any u € n we have ker{u) 5 Vq and uM^" QVq® M^"*. 

Let Xg be an element in Os C 5p2m(^)- Fix an arbitrary u E n such that Xs + u G O^- Then 
Xs + u & Oj) for some D. Our main task is to determine all possible D. Since the signs for odd 
rows of D are prefixed, we only need to determine the signs for even rows of D. 

Let {H, Xs + u, Y} be a standard triple. Then R^" decomposes into a direct sum of subspaces 

Fi © © • • • © V; © Vr+l © ... © Vn-m, 

such that 

• dim Vi = di = Si + 2 for i E [1, r] and dim Vi = 2 for i > r. 

• For even di, r^jy. is nondegenerate. 

• For an odd pair di = dj+i, r2|vj®v,+i is nondegenerate, Vi and V^+i are both isotropic. 

• All Vi for even di and Vi © V^+i for (dj , ) an odd pair, are perpendicular to each other 
with respect to the symplectic form Q. In other words, 

^ = [®i even^Wi] © [®{di,d,+i) an odd pair^\v,iSV,+ i]- 
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• Xs + u decomposes accordingly as 



Xi®X2®...eXre Xr+l . . . e Xn^m, 

where Xi is a principal nilpotent element in sp(Vi, illvj for even dimensional Vi and Xj©Xj+i 
is a nilpotent element in 0,+ C sp(Vi © Vi+i, for the odd pair (dj = dj+i). 

This is essentially the abstract version of Jordan decomposition for a nilpotent element in a real sym- 
plectic Lie algebra. For even dimensional Vi, there are two principal nilpotent classes parametrized 
by df and respectively. We now need to determine the sign attached to Xi for each even dj. 

Let di be an even integer. Generally speaking, the sign attached to Xi can be determined as 
follows. There exists a basis of Vi 

r (l) (2) (dOl 
\C-i , Cj ' • • • ! "^i J 

such that 

Xep) = ep) (ViG[l,d.-l]), 
and Xicf^^^ = 0. Then the signature of 

gives the sign attached to di for Xi (see Page 139 |CMj ). For the standard triple {h, x, y} C s[2(M), 
Ox will correspond to 2"*" and O-x will correspond to 2~ . 



Suppose that is an even integer greater than 2. Now we need to connect the signature attached 
to Xi with its "projection "to M^™. Notice that XsVq = and uVq = 0. So Vq C /cer(Xs + u). 
Since Xs + n E Od , the kernel of Xs + n is of dimension n — m which is equal to the dimension of 
Vq. We see that ker{Xs + u) = Vq. Hence 

{Xs + n)(M2-) = = Vq® M^"^. 

In particular, 

span{ef , . . . , = Xi{Vi) = Vir^ (Vb M^™). 
For j G [2, di], put ep^ = f^^^ + up'^ where //^'^ G M^m ^^^^ ^(i) ^ j^g^^g ^ [2, - 1], 

(Xs + u){fl^^ + = (Xs + u)^^ = X,(ep)) = ep+^) = + 
Since n(R2™ © y^) c Vq and XsVb = 0, we have 

Xsf? = fl'^'^, ne^^=vt'^. 

Let Wi = spanj/.^'^^ | j G [2,dj — 1]}. Notice that Q restricted to span{e.^\ . . . ,ef^^ is nonde- 
generate. Since Vq is isotropic and Wi _L Vq, we see that il. restricted onto Wi is nondegenerate. 
Furthermore, Wi are mutually perpendicular with respect to O since 14 's are perpendicular to each 
other. So XsIh/; is a Jordan block of size di — 2 = Si. The sign attached to it, is the signature of 

n{x^.^-'f\fy) = n{ixs + ur-'ef\ef^). 
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Observe that 



n{{Xs + ur-'e)'',er') = n{{Xs + ureY',{Xs+u)e\'') = -n{{Xs + u, , 

Hence the sign attached to di for Xs + u differs from the sign attached to Sj for Xs- (3) is proved. 
We have shown that 

Lastly, we need to show that each Oj-^u) appears in 

Kfe-^i.M)e.,..(M){0} X Os. 

It suffices to construct an element Xj-f(j) G l^s+n for an element Is G Os- Let {ei, 62, ... , en-m} be a 
basis for Vq and {cn-m+i, ■ ■ ■ , &2n-2m} be the dual basis for Vq such that the natural bilinear 

form coincides with the restriction of onto Vq (8> Vq- Fix a standard basis {e2n-2m+i, ■ ■ ■ ■, &2n} for 
(R^"*, fi|K2m). Let Ei^j be the 2n x 2n matrix with (z,_7)-th entry equal to 1 and the rest equal to 
zero. Without loss of generality, suppose that 

Ys=Yi®Y2® ...Yr 

is a Jordan decomposition such that 

• For even Sj, 

£i. 1 

2 

k=l 

for some jj > 2n — 2m; 

• For a pair of odd integer (sj = Sj+i), 

Si-l Si-1 

fc=l k=l 

for some jj > 2n — 2m. 
For even Sj, define 

= + ^ ~ Em+ji+l,n-m+i- 

For odd pair (s, = Sj+i), define 

For i G [r + 1, n — m], define = ^Ei^n-m+i such that the number of — is j and number of + is 
n — m — r — j. Let X = J2i=i^ ^i- I* is then easy to check that X G OdO) • □• 

Our method can be generalized to compute induced real nilpotent orbits. We shall do this in 
a subsequent paper. 
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Corollary 2.3 Let t = [1"]. Then 

gl(n,IR) ^ 

is the union of n + 1 orbits consisting of all Oj^ij) with d^^^ = [2"]. 

Theorem 2.8 Let Os be a nilpotent orbit of Sp2mi^) ■ Let s = (si > S2 > . . . > Sf > 0). Suppose 
n — m > r . Then 

is a union of nilpotent orbits Oj-^o) and each D^-'^ can be obtained by 

1. merging two copies of [l"-™-] to S from the left; 

2. extending the signs ofS; 

3. filling the n — m — r rows of length 2 in D^-'-' with j I " l+l and n — m — r — j l+l ' I . 

Sketch of the proof: Recall that t{Os) = C>t{S) ^^'^ is obtained by switching the signs of even 
rows in S. Suppose that sf represents an even row in S. Then s^ represents the corresponding 
even row in t(S). By Theorem 12.71 df^ will represent the corresponding even row in D(J'). This IS 
the same as adding two boxes to the even rows of S and extend the signs of S. (3) follows directly 
from Thm. O □ 
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Chapter 3 

Theta Correspondences and Quantum 
Induction 

Notation 13 Let Mp2ni^) be the metaplectic covering of Sp2ni^)- Let {l,e} he the preimage of 
1 € Sp2n(^)- For any subgroup G of Sp2n(M), let MG be the preimage of G under the metaplectic 
covering. 

Fix a dual pair {0{p, q) , Sp2n{^)) in Sp2n{p+q){^) (see |Ho79j ). Fix a maximal compact subgroup 
K for Mp2n{p+q){^) such that K n MO{p,q) is a maximal compact subgroup of MO{p,q) and 
K n MSp2ni^) is a maximal compact subgroup of MSp2ni^)- Let uj{p,q;2n) be the oscillator 
representation of the group Mp2n{p+q)(M)- If the parameters (p, g;2n) are apparent, we will write 
u!. Let V{co) be the Harish-Chandra module. Unless otherwise stated, any representation vr of 
MG in this paper satisfies 7r(e) = —1. Very often, this assumption reduces our discussion and 
computation to representations of the linear group. 

Notation 14 Let G be any reductive subgroup of Sp. Let 7l{MG,io) be the equivalence classes of 
irreducible Harish-Chandra modules of MG that occur as quotients ofVi^uj). 

Theorem 3.1 (Howe, |Ho89j ) Let (Gi,G2) be a dual pair in Sp. Then Tl{MGiG2,i^) yields a 
one-to-one correspondence between TZ{MGi,u;) and TZ{MG2,uj). 

This correspondence is often called (local) theta correspondence, Howe's correspondence or duality 
correspondence. Let 9{p, q; 2n) be the theta correspondence from TZ{MO{p, q),u)) to TZ{MSp2n(M),'-^) 
. Let 6{2n;p,q) be its inverse. If the parameters {p,q; 2n) are apparent, we will just write 6. The 
description of the sets TZ{MO{p,q),iv) and Tl{MSp2nO^),^) is not known in general. 

If G is a subgroup of Sp, we say that MG splits if MG ~ G x {±1}. For the dual pair 
{0{p,q), Sp2nO^)), MO{p,q) always splits and MSp2nO^) splits if and only ii p -\- q is even. If 
p + g is odd, MSp2nO^) is isomorphic to the metaplectic group Mp2nO^)- In all cases, 9 can be 
regarded as a one-to-one correspondence between a certain subset of Il{Mp2ni^)) ^nd a certain 
subset of n(0(p, q)). Since this is the viewpoint in some literature (for example [KR| ) . we will also 
take this viewpoint if necessary. 
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3.1 Theta Correspondence in Semistable Range 

We adopt the notation from (1.3.1) and (1.3.2). Let n be the constant vector (n,n, n, . . . ,n) of a 
fixed dimension. 

Definition 16 (Def. 5.1 [HeOOj . Def. 3.2.1 |Heq] , Thm. 8.17, 8.18, Def. 8.3, 8.4 |He98] l 

An irreducible representation tt of MO{p, q) is said to be in the semistable range of 9{p, q\ 2n) if 
and only if every leading exponent v of tt satisfies 

^{v) -n + 2p{0{p,q)) -< 0. 

An irreducible representation vr of MSp2n{^) is said to be in the semistable range of 9{2n;p,q) if 
and only if every leading exponent v of tt satisfies 

5^W-^^ + 2p(5p2n(M)) ^0. 

We denote the semistable ranges by Rs{MO{p, q),u!) and Rs{M Sp2nO^) , ^) respectively. The reader 
should note that a representation vr in Rs{G,uj) may not occur in 1Z{G,lo). This is pointed out to 
me by Chen-Bo Zhu. 

Definition 17 ( [Ll89], [HeOO]) Consider {Gi,G 2) = {0{p,q), Sp2n{^)) or (^1,^2) = {Sp2n(^),0{p 
Let IT G Rs{MGi,uj). Define a bilinear form ( , )n on V{lo) ® ViT^^) 

{(t)®u,ip®v)T,= I {uj{g)(t),ip){v,7r{g)u)dg {(t>,ip e V{uj),u,v G V{Tr)). 
JMGi 

Let TZtt be the radical of { , )7r. Define 

e,iMGi,MG2)i7T) = Viij) <S) V{Tr^)/n^. 

9s{MGi, MG2){tt) inherits an infinitesimal MG2 action from lo{MGi, MG2)- It is a (g, ii')-module 
of MG2. 

Theorem 3.2 (Theorem 1.1 [HeOOj , [Heu] ) Suppose n is a unitarizable Harish- Chandra mod- 
ule in the semistable range of 9(MGi, MG2)- Then ( , )n is well-defined. If { , )7r 7^ 0, then 
9s{MGi, MG2){tt) is an irreducible Hermitian Harish- Chandra module of MG2 and 9s{MGi, MG2){tt) 
is equivalent to 9{MGi, MG2){tt). 

This theorem basically says that if ( , )7r is well-defined and nonvanishing, then 9s is Howe's corre- 
spondence ( |Ho89j ) on the Harish-Chandra module level. By Howe's Theorem, 9s{MGi, MG2){tt) 
is an irreducible Harish-Chandra module of MG2. The notation ( , ),r is essentially due to Jian-Shu 
Li f p89] ^. 

In [HeOlj . we proved the following nonvanishing theorem. 

Theorem 3.3 (Corollary 1.1 |He0 1]) Suppose p + g < 2n + 1. Suppose it € Rs{MO{p, q),u}). 
Let det be the lift of the determinant ofO{p,q) to MO{p,q), i.e., 

ker(det) = MSO{p,q). 

Then either 9s{p, q; 2n)(7r) 7^ or 9s{p, q; 2n)(7r (g) det) ^ 0. 
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3.2 Unitarity and Strongly Semistable Range 

In |Heuj ■ we proved the unitarity of 9s{tt) for tt in the strongly semistable range. 

Definition 18 An irreducible admissible representation vr is in Rss{MO{p,q),uj) if every leading 
exponent of tt satisfies 

^{v) - (n - Pi^) + p{0{p, q))^0. (3.1) 

An irreducible admissible representation vr of MSp2nO^) ^-^ Rss{MSp2nO^),^) if every leading 
exponent of tt satisfies 

^{v)-{^^^-ii-l)+p{Sp2nm)^0. (3.2) 
We call Rss the strongly semistable range. 

Notice that i2s(M5p2n(K), w) and Rss{MO{p,q),uj) only depend on p + q, not on a particular pair 
(p, q) . Since 

^iv) - n + 2p{0{p, q)) -< ^{v) - (n - P±^) + p{0{p, q)) 

and 

^(V) - + 2p(5p2n(M)) ^ K(t;) - (^i-5 - n - 1) + /9(5p2n(K)), 

by the definition of semistable range, Rss Q Rs- Thus 6s in the strongly semistable range is the 
same as the original 9. 

Theorem 3.4 (Corollary 5.2 f Heu] ) Suppose 

• p + q < 2n + 1; 

• 7re RssiMO{p,q),uj); 

• IT is unitary; 

Then { , is positive semidefinite. If 9s{p,q]2n){TT) ^ 0, then 9s{p, q;2n){TT) is an irreducible 
unitary representation of MSp2n(^)■ 
Theoreul 3.5 (Corollary 5.1 [HeuJ ) Suppose 

• n < p < q 

• ne Rss{MSp2n{^),u;); 

• TT is unitary; 

Then ( , )^ is positive semidefinite. If 9s{2n]p,q){TT) ^ 0, then 9s{2n;p, q){Tr) is an irreducible 
unitary representation of AIO{p,q). 

Strictly speaking, 9s{*){tt) is an irreducible unitarizable Harish-Chandra module. Notice that ( , )7r 
can be regarded as an inner product on 9s{*){it). We can simply complete ( , )7r to obtain the 
Hilbert space of 9s{*){tt). In general, invariant inner product on an irreducible Harish-Chandra 
module is unique up to the multiplication of a constant. 
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Theorem 13.41 and Theorem 13.51 provide us the basis to construct unitary representation through 
the theta correspondence in the strongly semistable range. 



3.3 Quantum Induction Q 

Notation 15 Suppose that p + q + p' + q' = (mod 2). Suppose that p' + q > 2n + 1 and 
q' + p > 2n + 1. Let £{p' + q,q' + p; 2n) be the representation 7r„ of 0{p' + q,q' + p) studied by 
Zhu-Huang in IZH^ . The representation £{p' + q,q' + p;2n) is essentially 9 {2n]p' + q,q' +p)(tiivial) 
restricted to the 0{p' + q,q' + p) component in MO{p' + q,q' + p)- 

Notice here that MO{p' + q,q' + p) sphts into 0{p' + q,q' + p) and 0{p' + q-,q' + p)e. 

Definition 19 Suppose 

p' + q > 2n + 1, p + q' > 2n + 1, p + q = p + q' (mod 2). 

Let 0{p, q) X 0{p', q') be a subgroup diagonally embedded in 0{p' + q, q' +p)- Fix a maximal compact 
subgroup K = 0{p' + q) x 0{q' + p) such that 



are maximal compact subgroups of 0{p,q) and 0{p',q') respectively. Let vr € li{0{p,q)) be such 
that 



converges absolutely for every ui,U2 G V{£{p' + q,q' +p;2n)) and every vi,V2 G Vi"^)- Let TZ be 
the radical of { , ) as a Hermitian form on V{£{p' + q,q' + p; 2n)) (g) V{-k). Then 



Q{p,q;2n;p' ,q'){Tr) is a {o{p' ,q'), K f] 0{p' ,q'))-m.odule. 

Conjecture 1 Suppose 

p' -\- q > 2n -\- 1, p + q' > 2n -\- 1, p + q = p' + q' (mod 2), p' -\- q' > p + q. 

Lf Q{p,q;2n;p' ,q'){Tr) ^ 0, then Q{p,q;2n;p',q'){TT) is an irreducible admissible representation of 
0{p',q'). If IT is unitary, then Q(p,q;2n;p' ,q'){TT) is also unitary. 

Notation 16 For p + q<n + n' + l, put £(2n + 2n';p, q) = 9{p, q; 2n + 2n') (trivial). 

p + q < n + n', £{2n + 2n';p,q) is unitary according to Howe-Li's theory on stable range dual 
pairs ( |Ho84] . |LI89] ). U p + q = n + n' + 1, £{2n + 2n';p, q) is unitary according to the result of 
Przebinda on almost stable range dual pairs (see Lemma 8.6 |PR93j ) or a Theorem of Kudla and 
Rallis ( [KRj ) . £{2n + 2n';p, q) is a genuine representation of Mp2n+2n'0^) p + q is odd. 



KnOip,q) 



KnO{p',q') 




{£{p' + g, q +p\ 2n){g)ui,U2){iT{g)vi,V2)dg 



iVi£ip' + q,q'+p; 2n)) ® y(^))/7^ 
inherits an infinitesimal 0{p' ,q')- action. Define 

Qip,q;2n;p',q'){Tr) = V{£{p' + q,q' + p-2n)) (^V{t:)/TI. 
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Definition 20 Suppose n + n' + l > p + q. Let Sp2ni^) x Sp2n'i^) be diagonally embedded into 
Sp2n+2n' ^) ■ Let K he a maximal compact subgroup of Sp2n+2n' (I^) such that 



Sp2nmr\K, Sp2n'(M)nK 

are maximal compact subgroups of Sp2n{^) o.nd Sp2n'{^) respectively. Let vr he an irreducible 
representation of Mp2n{^) such that the following Hermitian form { , ) on V{£{2n + 2n';p,q)) 
V{tt) converges: 

{ip(g)u,'i(g)v) = / {£{2n + 2n';p,q){g)ip,<;){Tr{g)u,v)dg 

JMSp2n{K) 

V ^,(; eV{£{2n + 2n;p,q));u,v eV{TT). 

Define Q{2n\p,q;2n'){Ti) to heV{£{2n\p,q;2n'))®V{'i:) modulo the radical of { , ). Q(2n;p, g; 2n')(7r) 
inherits an infinitesimal 5p2n'i^)- action from £{2n + 2n';p,q). It is a {5p2n'(M), f^Mp2n'(M)))- 
module. 



Conjecture 2 Suppose n+n'+l > p+q and n' > n. If Q{2n;p, q; 2n')(7r) ^ 0, then Q(2n;p, q; 2n')(7r) 
is an irreducible admissible representation of Mp2n'0^)- If is unitary, then Q{2n;p,q;2n'){7T) is 
also unitary. 



Q(*) defines an induction process. We call Q{*) quantum induction. 



3.4 Unitary Quantum Induction Q 

Q(*) is closely related to a composition of two Og- The way it is defined allows us to bypass several 
technicalities which we shall address in this section. One difficult problem arising from composing 



two 6s is that the range of one 6^ may not be in the domain of the other Bg- In Heq , we study 
the leading exponents of 0s(*)(7r). Under certain assumptions, one can compose Og with another 
Og. Surprisingly, the strongly semistable range plays a crucial role in the composability of Og- By 
identifying Q with a certain composition of Og, we established the unitarity and irreducibility of Q{it) 
for vr in the strongly semistable range. Unitary quantum induction then enables us to construct 
certain irreducible unitary representations whose existence has been conjectured by Arthur and 
Barbasch-Vogan. 



Theorem 3.6 (Theorem 7.1.1 [HeqJ ) Let vr he an irreducible unitary representation in 

Rgg{MO{p, q),io{p,q;2n)). 

Suppose 

• q' > p' > n; 

• p' + q' - 2n > 2n - {p + q) + 2 > 1; 

• p + q = p' + q' (mod 2). 
Then 
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1. es{p,q;2n){7r) G i2,,(M5p2n(M), g'; 2n)); 

2. Q{p,q;2n;p',q'){Tr) ^ es{2n;p',q')es{p,q;2n){Try, 

3- If Q{p, q',2n;p' , q'){'ir) ^0, then Q{p,q;2n;p' ,q'){TT) is unitary. 

Let me say a few words about the proof of this theorem. (1) is proved in |Heq| through an estimate 
on the matrix coefficients of 9s{p,q;2n){TT). By (1), 9s{2n;p' ,q')6s{p, q;2n){7r) is weh-defined. In 
fact, by Definition \T7\ the Harish-Chandra module of 9s{p,q',2n){7r) consists of "distributions" of 
the following form 

/ uj{p,q;2n){gi)(l)(S)7r''{gi)udgi {(p £ V{u;{p,q;2n)),u £ V{7r)). 

JMO(p,q) 

For the same reason, the Harish-Chandra module of 9s{2n;p' ,q')6s{p,q;2n){TT) consists of "distri- 
butions" of the following form 

[ u;{p',q';2n)ig2)iJ^es{p,q;2n){TTy{g2)vdg2 (V G V{u;{p' ,q';2n)),v G g; 2n)(7r))). 

jMSp2„m 

Combining these two statements, V{6s{2n;p' ,q')9s{p,q;2n){-7T)) consists of "distributions" of the 
form 



uj{p ,q ■,2n){g2)tp (g)uj{p,q;2ny{g2gi)4>^T^{gi)udgi dg2 (3.3) 

92eA/5p2n(ffi) JgieMO(p,q) 

(V- G V{L0{p',q';2n)),cl)€ V{u;{p,q;2n)),u G V{7r)). 
Notice that q; 2nY = uj{q,p; 2n) and 

uj{p' , q'; 2n) (g) uj{q,p; 2n) = uj{p' + (?,(/'+ p] 2n). 
So Equation ( 13. 3|) becomes 

/ / ^{p' + q,q' +P;2n){i{gi)g2){ip®4>)<»TT{gi)udgidg2. 

Jg2&MSp2nO&) Jgi(^MO{p,q) 

Here i : — > i{gi) is a canonical embedding of MO{p, q) into MO{p' + q,q' + p)- By the theorems 
of Howe-Li, 

{ / uj{p' + q,q' +p;2n){g2){'>p(S)^)dg2 \ %b G V{uj{p' ,q';2n)),^ e V{Lo{q,p;2n))} 

Jg2eMSp2nm 

can be identified with the Harish-Chandra module of £{p' + q, q' + p; 2n). Therefore, 

V{es{2n-p\q')es{p,q;2n){^)) 

consists of 

/ E{p' + q,q' +p;2n){i{gi))r]®-K{gi)udgi (r? G V{8{p' + q,q' + p;2n)),u V{tt)). 

JMO(p,q) 



35 



We have assumed that 7r(e) = —1 from the beginning. The integral over MO{p,q) is just twice 
the integral over 0{p,q). It follows that V{9s{2n; p' , q')6s{p, q;2n){TT)) consists of vectors of the 
following form 

/ £{p' + q,q' +p;2n){i{gi))7]®TT{gi)udgi [r] G V{£{p' + q, q' + p;2n)),u e V{tt)). 
Jo{p,q) 

The absolute convergences proved in |Heq| allow us to change the order of the integrals. Thus 

esi2n;p',q')esip,q;2n)i7r) ^ Q(p, g; 2n;p', (?')(vr). 
(3) follows easily by Theorems 13.41 and 13.51 □ 
Definition 21 Under the hypotheses in Theorem 13.61 define 

Qip, q; 2n; p', q'){7r) = 6, (2n; / , q')e, {p, q; 2n) (^) . 

We call Q unitary quantum induction. By Theorem 13.61 Q{p,q;2n;p' ,q'){TT) is well-defined and 
equivalent to Q{p,q;2n;p' ,q'){Tr). A similar statement holds for Q{2n;p,q;2n'){iT). 

Theorem 3.7 (Theorem 7.2.1 [Heq] ) Letn be a unitary representation in Rss{MSp2n{^),^^{p,Q 
Suppose 

• 2n' — p — q>p + q — 2n — 2; 

• n < p < q. 

Then 

1. 9s{2n;p,q){7T) e Rss{M0{p,q),u;{p,q;2n')y, 

2. Q{2n;p,q;2n'){TT) ^ 9s{p,q;2n')9s{2n;p,q){7T); 

3. if Q{2n;p,q;2n'){Tr) ^ 0, then Q(2n;p, g; 2n')(7r) is unitary. 

Definition 22 Under the hypotheses of Theorem \3. 7| , define 

Q{2n-p,q;2n'){'K) = 9s{p,q;2n)esi2n;p,q)(7r). 

By Theorem 13.71 Q(2n;p, g; 2n')(7r) is well-defined and equivalent to Q{2n;p,q;2n'){Tr). 

In summary, Q is a generalization of Q = 9s o 9s. Without the requirement of unitarity, we 
can loose some restrictions posed in Theorems 13.61 and 13.71 We only state the Theorem for 
9s{p,q',2n')9s{2n;p,q) since we will need it in the next Chapter. 

Theorem 3.8 Let ir be a unitary representation in Rss{MSp2n(^)^^{p^(l\2n)). Suppose 2n' — 
p — q>p + q — 2n — 2. Then 9s{2n;p,q){TT) G RssiM0{p,q),u;{p,q;2n')); and 

Q{2n; p, q; 2n') (^) ^ 9s {p, q; 2n')9s (2n; p, q) (vr) . 
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This theorem follows directly from the estimate established in Theorem 6.3.1 and Definition [THlof 
|Heq| . The proof is omitted. 

The advantage of introducing Q will be manifested when we begin to relate unitary quantum 
induction to parabolic induction. Based on the studies on parabolic induction, we will establish 
the nonvanishing of 9s{2n;p,q){TT) under some restrictions. The nonvanishing of 9s{2n;p,q){Ti) is 
otherwise very hard to establish if p or q is less than 2n. 

3.5 Moment Map and 6(^1, ^2) 

From now on, let Gi = 0{p, q) and G2 = Sp2n(M)- Write 

Gic = 0{p + q,C), G2C = Sp2n{C). 

Let Mat{p + q; 2n) be the set oi p + qhy 2n real matrices. Let 

Definition 23 Define the moment maps 

mi : X E Mat(p + q, 2n) — > Ip^gxWnX^ E o{p, q), 

1712 : X e Mat(p + q, 2n) WnX^Ip^qX E 5p2n(I^)- 

We say that x is nilpotent if mi[x) is nilpotent. 

Observe that mi(x)' = /p_qXm2(x)'~^VF„x*. We have 

Lemma 3.5.1 mi[x) is nilpotent if and only if m2{x) is nilpotent. 

The group Sp2nO^) x 0{p, q) acts on the set of nilpotent element in Mat(p + (7, 2n). Thus we obtain 
a set of nilpotent Sp2n^) x 0(p, g)-orbits in Mat(p + q,2n). Clearly, rui induces a map from the 
nilpotent orbits in Mat(|? + q, 2n) to nilpotent orbits in g^. 

Similarly, we define the complex moment maps, still denoted by mj. This should not cause any 
notational problem. For example, the usage of D usually points to the real moment map and the 
usage of d usually points to the complex moment map. 

For a complex nilpotent orbit Oi in g^^, consider the closure of 

7712 (rri]^^ (Closure of Oi)). 
By a Theorem of Daszkiewicz-Kraskiewicz-Przebinda, the closure of 

1712(171^^ {Closure of Oi)) 

is the closure of a unique nilpotent orbit in (see |DKPj ). This yields a correspondence between 
certain nilpotent orbits in gi^ and certain nilpotent orbits in g2c- We denote this correspondence 
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by 6(Gic, G2c)- Similarly, we define Q{G2Cj Cic). The reader should be warned that 0(Gic, G2c) 
is NOT the inverse of 0(G2c,G'ic). For the nilpotent orbits we are concerned with, 0(Gic,G2c) 
and @{G2c, Giq) are quite easy to describe ( |DKPj). 

Lemma 3.5.2 (see Thm. 4.2 [DKP]) Let Od ^ U he a nilpotent orbit of either 0{p,q) or 
Sp2ni^) (see Definition \12\) . Construct an alternating sequence of complex orthogonal orbits and 
complex symplectic orbits 

Od, Od-l,Od-2, ■ ■ ■ Od-di+l 
and a corresponding alternating sequence of complex orthogonal groups and symplectic groups: 

G{Od),G{Od-i),...,GiOd-d,+i). 

Then V j, 

e(G(Od-j),G(Od-j+i))(cZ(Od-j)) = cliOd-i+i). 

We define G(G2, Gi) and 0(Gi, G2) for real nilpotent orbits in the same fashion. Pan showed that 
0(G2,Gi)(Od) is the closure of at most two real nilpotent orbits (see Theorem 8.11 |PANj ) . We 
give two lemmas that can be easily deduced from the descriptions of 0(G2,Gi) and 0(Gi,G2) in 

Lemma 3.5.3 (see 6.4, 8.6 [PAN] ) Suppose p + q < 2n. Then the real nilpotent orbit Od 
occurs in the image 0/7712 if md only «/G(Od-i) = 0{r,s) with r <p and s < q. 

Lemma 3.5.4 (see 6.4, 8.1, 8.11 fPAN] ) Let On E U be a real nilpotent orbit of either 0{p,q) 
or Sp2ni^)- Construct an alternating sequence of real nilpotent orbits of symplectic groups and 
orthogonal groups 

CD) C'd-I, 00-2; • • • C'D-di+l 

and a corresponding sequence of real groups 

G(Od), G(Od-i), G(Od-2), . . . G(OD-di+i). 

Then V j, 

e(G(c:)D-j),G'(OD^+i))(c/(OD-j)) = c/(OD-j+i). 
3.6 Przebinda's Results on V{Ann 0{t:)) 

In |PR93| ■ Przebinda proved that under some conditions, the following diagram commutes: 

TT e{MG,MG'){TT) 

(3.4) 

ViAnmr) ^^^^'^"^^ V{Ann e(MG, MG'){7r)) 

(See Notation 5). We state Przebinda's theorem for 0(p,q) and Mp2ni^)- Recall from Theorem 
III.l jMlj and Theorem 8.48 of [KNj that vr has the rate of growth 7 if and only if for every leading 
exponent v of n, 

mv)^{j-l)p{G) 

(see also Theorem 4.5 of [PR93]). 
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Theorem 3.9 (Theorem 0.9 [PR93| ) Letn be an irreducible unitary representation ofO{p,q). 
Suppose 

1. 2n+l>p + q; 

2. has rate of growth 7 with 7 + 1 < p^g_2 > equivalently, every leading exponent v of ir 
satisfies 

'?n 

p + q- 2 

3- 7 > 0, equivalently, 7 is not a discrete series representation; 

4- ( , converges and does not vanish; 

5. { , )tt is positive semidefinite; 

6. there exists a full rank element x G Mat(p + q,2n) such that the 0{p,q)-orbit generated by 
mi(x) is of maximal dimension in AS{QTr). 

Then 9{p,q;2n){TT) is unitary and its associated variety is the complex orbit 

Q{0{p + q, C), Sp2n{C)){V{Ann t,)). 

Let me make two remarks here. Firstly, the assumption (1) does not appear in Przebinda's original 
theorem. Conditions (2) and (3) imply 2n > p + q — 2. We add (1) as an assumption for the sake 
of clarity. The reader should also notice that condition (1) guarantees the nonvanishing of ( , 
or ( , )det (giTT by Theorem 13.31 Secondly, condition (6) is satisfied in the setting of Lemma 13.5.41 
More precisely. 

Lemma 3.6.1 Suppose that vr is an irreducible unitary representation of 0{p,q), that 2n + 1 > 
p+q, and that T) is a signed Young tableau for Sp2nO^) ■ Assume that Od-i is of maximal dimension 
in AS{@Tt). Then there is an element x £ Mat(p + q, 2n) of rank p + q with the properties 

1. 0{p,q)mi{x) = Od-i; 

2. 5p2nWm2(x) = Od. 

Proof: According to Lemma 13.5.31 Od occurs in the image of m2. By Lemma 13.5.41 there exists 
X in Mat(p + q, 2n) such that m2{x) G Od and mi{x) G cI{Oy)-i). Hence 

rank{WnX* Ip^qx) = rank{m2{x)) = p + q. 

But x G Mat(p + q, 2n). So x must be of full rank. For x of full rank, observe that 

[m2{x)Y' = WnX^[mi{x)Y-^Ip^gX. 

By writing out the defining equations for m2{x) G On, we obtain mi{x) G Od-i- But mi{x) G 
cl{Ou-i)- It follows that mi{x) G Od-i- □ 

Similarly, Przebinda's theorem for Mp2nO^) can be formulated as follows. 
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Theorem 3.10 (Theorem 0.9 [PR 93]) Letir be an irreducible unitary representation of Mp2ni^)- 
Suppose 

1. p + q > 2n; 

2. has the rate of growth 7 with 7 + 1 < equivalently, every leading exponent v of tt 
satisfies 

m -< - 2)p(Mp2n(M)); 

3. 7 > 0, equivalently, vr is not a discrete series representation; 
4- ( , converges and does not vanish; 

5. ( , ),r is positive semidefinite; 

6. there exists a full rank element x £ Mat(p + (;,2n) such that the Sp2n{^)-orbit generated from 
m2{x) is of maximal dimension in AS{Qt^). 

Then 9{2n;p,q){TT) is unitary and its associated variety is the complex nilpotent orbit 

e{Sp2n{C), 0{p + q, C)){V{Ann vr)). 

It is easy to see that (2) and (3) imply (1). We leave (1) there for the sake of clarity. Similar to 
Lemma 13.6.1^ we have 

Lemma 3.6.2 Suppose that vr is an irreducible unitary representation of Mp2ni^), thatp+q > 2n, 
and that T) is a signed Young tableau for 0{p,q). Assume that Od-i is of maximal dimension in 
AS{Qt^). Then there is an element x G Mat(p + q, 2n) of rank 2n with the properties 

1. 0{p,q)mi{x) = Ou; 

2. 5p2nW"l2(x) =Od-1. 

So (6) in Theorem 13. 101 holds if there is an orbit in AS{Qt^^ of the form Od-i for D G yD{p, q). 
Last let us recall the following result of Przebinda. 

Theorem 3.11 (Corollary 2.8, |PR93j ) Consider the dual pair {0{p,q), Sp2ni^))■ 
• If it e TZ{MO{p,q),uj), then WF{tt) is a subset of mi {Mat {p + q;2n)). 
• Ifne n{MSp2„(M),uj), then WFiir) is a subset of m2{Mat{p + q;2n)). 
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Chapter 4 

A Nonvanishing Theorem 



In [HeOlj . we established a nonvanishing theorem for 9s{p, q; 2n) with p+q < 2n+l. This theorem is 
cited as Theorem 13.31 in the last Chapter. In this chapter, we will prove a nonvanishing theorem for 
6s{2n;p, q). These two nonvanishing theorems will then be used to construct a packet of irreducible 
unitary representations N{0) attached to a real special rigid orbit O and more generally any O (zU 
(see Definition [T2]) . We start with a trivial lemma. 

Lemma 4.0.3 Suppose that 

9sip, q;2n2)0si'2ni;p, q){7r) = Q{2ni;p, q;2n2)iir). 

IfQ{2ni;p,q;2n2){TT)^0, then ^,(2ni;p, / 0. 

Combining with Theorem 13.81 we obtain 

Theorem 4.1 Suppose 2n2—p—q > p+q—2ni—2. Letir be an irreducible admissible representation 
in Rss{MSp2m{^),oj{p,q;2ni)) (see Def. \M- // Q(2ni;p, g; 2n2)(7r) / 0, i/ien 0,(2ni;p, g)(7r) / 
0. 

At first glance, Q(2ni;p, g; 2n2)(vr) seems to be more difficult to treat than 6s{2ni;p,q). This is 
true with one exception. As predicted in [Heq] , when p + q = ni+n2 + l, quantum induction 
can be obtained from parabolic induction. In this chapter, we will examine Q(2ni;p, g; 2n2)(7r) 
for ni + n2 + 1 = P + q- We will prove that precisely for ni + n2 + 1 = p + q and ni < n2, 
Q(2ni;p, q; 2n2)(7r) can be obtained from parabolically induced representations. As a consequence, 
we can determine the nonvanishing of Q(2ni;p, (7; 2n2)(vr). 

Notation 17 Let m,n be two nonnegative integers such that m + n 7^ 0. Let Pm,n be the standard 
maximal parabolic subgroup of Sp2m+2n{^) with Levi subgroup GL{m,M) x Sp2ni^)- 

4.1 Nonvanishing of Q{2ni;p,q; 2{p + q — rii — l))(7r): Overview and 
Main Result 

Suppose that 77-2 > ni. Let Sp2ni (M) x Sp2n2 0^) be a subgroup of Sp2ni+2n2 0^)- Let U{ni+n2) be a 
maximal compact subgroup of S'p2ni+2n2 (I^) such at U{ni+n2)riSp2ni(M) and U{ni+n2)r\Sp2n2(M) 
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are maximal compact subgroups of 5p2ni (1^) and Sp2n2 respectively. 



First of all, a theorem of Kudla and Rallis says that, for some a, the parabolic induced repre- 
sentation 

r = IndS"''''"'^''^x" 

(see Definition decomposes into a direct sum of irreducible representations, namely 8{2ni + 
2n2]p,q) for some p and q ( |KR| . |LZ| ). Regard /" as a representation of Mp2ni(K) x Mp2n2(I^) 
by restriction. For vr G n(Mp2rai(K)) satisfying a certain growth condition, induces a represen- 
tation of Mp2n2(I^)> -^"(tt) (see Definition [25]) . Using the theorem of Kudla and Rallis, we show 
that /"(vr) decomposes into a direct sum of Q{2ni;p, q; 2n2)(vr) for some p + q = ni + n2 + 1- 

Next, we study the restriction of to Mp2ni(I^) x Mp2n2i^)- Notice that the Harish-Chandra 
module V{I'^) consists of sections of a homogeneous line bundle over the Lagrangian Grassmannian 
Q£]^2ni+2n2_ gy analyzing the action of Mp2ni{^) x Mp2n2{^) on the Grassmanian, we prove that 



whenever the left hand side is well-defined (see Theorem 14. lip . In order that /"(vr) is well-defined, 
vr must satisfy a certain growth condition. The details are given in Section 14. 4[ 

Now, let us assume that /"(vr) is well-defined. Then we know that 



Ind7pl7_!,,„,^' =^ Q{2nv,P, q; 2n2)(7r) 



for a set of pairs {p,q)- The remaining question is which Q(2ni;p, g; 2n2)(7r) is nonvanishing. This 
turns out to be a hard question. We don't have any answer in general. But for some tt, we can 
detect the nonvanishing of Q(2ni;p, 2n2)(vr) precisely, thanks to the notion of wave front sets. 
Notice that the decomposition of /"(tt) into Q(*)(vr) induces a decomposition on the wave front 
level, namely, 

WF{IndZ'j^:^-^^l„^7r^ ® X") = U WF{Q{2n,;p, q; 2n2)(^)). (4.1) 
On the one hand, we know that 

V A'lt'„2-ni,ni J g[(n2 -ni ) (R) X Sp2„j^ (K) \ ^ 

As we have pointed out earlier, the induced orbit consists of a few irreducible components which 
can be parametrized easily. For the vr's we are concerned with. Theorems 12.71 and 12.81 provide us 
with the precise information about the nilpotent orbits contained in the left hand side of Equation 
KT\ 

On the other hand, for some tt, for example, those specified in Theorem 13.81 the right hand 
side of Equation 14.11 equals 

IJ WF{es{p, q; 2n2)es{2nr,p, q){7r)). 

Theorem 13.111 of Przebinda gives some control on which nilpotent orbit occurs in the wave front 
set of 9{p,q;2n2){*) ( |PR93| ) . Lemma 13.5.31 of Pan gives the parametrization of these nilpotent 
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orbits f [PXN] 1. 



We have now two sets of nilpotent orbits, one from the induced orbit, the other from the mo- 
ment map 7712 with respect to various {0{p,q), Sp2n2(M))- By matching these two sets of wave 
front sets, we obtain a nonvanishing theorem for W F {Q{2ni; p, q; 2n2)), which in turn gives us the 
nonvanishing of Q{2ni;p, q; 2n2) and 9s{2ni;p, q). Some of the results here will be used in Chapter 
6. In this Chapter, we will prove the following generic nonvanishing theorem. 

Theorem 4.2 Consider the group Mp2ni+2n2(^) with ni < n2- Let ir be a unitary representation 
in Rss{Mp2ni{^) , ^^{ni + n2 + 1,0; 2ni)) (see Def. \18\) . Let Od be a nilpotent orbit of maximal 
dimension in 

Lets be the subset of{{p,q) \ p+q = ni+n2 + l,p fixed parity}, consisting of those {p,q) such that 
On is contained the image of the moment map m2 associated with {0{p, q) , Sp2n2 0^)) ■ Then there 
exists a pair {p,q) G S such that Q(2ni;p, g; 2n2)('/r) ^ 0. For such a pair, 9s{2ni;p, q){-K) ^ 0. 



Notice that 

RssiMp2n, (M), u;(ni + n2 + 1, 0; 2ni)) = i?,,(Mp2ni (M), a;(p, q; 2ni)) 

(see Def. \T8\i for every p + q = ni + n2 + 1- The most favorable situation is when the set S, which 
depends on D, consists of only one pair {p,q). By our theorem, we will have 6s{2ni;p,q){TT) ^ 0. 
The proof of Theorem 14.21 will be given in Ch. 4.5. 



In this chapter, we take Q and Qs as a correspondence between representations of Mp2n(]^) and 
representations of 0{p,q). Throughout this Chapter, we assume n2 > ni. 



4.2 Results of Kudla-Rallis and Lee-Zhu 

Consider the symplectic group Sp2n{^)- Let MAN be the Langlands decomposition of Pnfl ( 
Notation [IT]). The Levi factor MA = GL{n,R). Let Mp2n(K) be the metaplectic covering of 
Sp2ni^)- The group MGL{n,M.) can be expressed as 

I X G GL(n,M),^2 ^ detx,C G C}. 

Put xiC^x) = Then x is a character of MGL{n,M.) of order 4. We extend x trivially on N. x 
becomes a character of MPnfl. 



Definition 24 1° = Ind^/p^^ 'X 



/" is a unitarily induced representation. 



Theorem 4.3 (Kudla-Rallis, [KRJ ) Suppose p + q = n + 1 and a = p — q (mod 4). Then 
0(p, 2n) (trivial) is a subrepresentation of I^. 



By studying the i^-types in 9{p, q; 2?7-) (trivial) for various p + q = n + 1, one has 
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Theorem 4.4 ( [KR], [LZ]) 

In= e(p,g;2n)(trivial). 

p+g=ra+l,Q=p— g (mod 4) 

More precisely, forn odd, 

ll = Indpf;^^Hrivial = e{p, q; 2n) (trivial), 

p+q=n+l,p=q (mod 4) 

ll = Indpf;^^^ sgn(det) = q; 2n)(trivial). 

p+q=n+l,p—q=2 (mod 4) 

For n even, 

Il= 0(^3, g;2n) (trivial), 

p+q=n+l,l=p—q (mod 4) 

Il= 0(^3,g;2n) (trivial). 

p+q=n+l,'i=p—q (mod 4) 

This decomposition theorem can be derived directly from the results in |KRj . It was explicitly 
stated in |LZ| . 

Notation 18 We use /" to denote -^"j+„2 which is the main object of study in this section. 

Definition 25 Let Sp2ni{^)^ Sp2n2(^) be embedded diagonally into Sp2ni+2n2(M)- Let U{ni+n2) 
be a maximal compact subgroup of Sp2ni+2n2(^) such at U{ni + n2) n Sp2ni(M) o,nd U{ni + 722) n 
Sp2n2 i^) o,"!"^ maximal compact subgroups of Sp2ni (^) CLnd Sp2n2 respectively. The embedding 
5p2ni(I^) Sp2ni+2n2{^) l^fts to Mp2ni(^) ^ Mp2ni+2n20^) (scc for example IHeiH^ ). Consider 
Let vr e n(Mp2ni(K)) such that the form ( , )i 

{4>i®ui,(l)2®U2)i= / {T^{g)ui,U2){L°'{g)(t)iA2)dg 
Jmp2„^{R) 

converges for every ui,U2 € V{TT),(j)i,(f)2 € V{L'^). Define /"(tt) to be V{L") (8) V{tt) modulo the 
radical of { , )i. /"(vr) is a (sp2„2(^)' -^^(^1 + "-2) Pi Mp2n2i^)) -module. 

From the definition of quantum induction Q and Theorem 14.41 we obtain 

Theorem 4.5 Suppose a = ni + n2 + 1 (mod 2). Then as (g, K)-modules 
r(7r)= Q(2ni;p,g;2n2)(7r) 

p+q=n\+n2+l,p—q=a (mod 4) 

whenever one side is well-defined. 

Let me remind the reader that 0(p, g; 2n) (trivial) is denoted by £{2n;p,q) in the definition of Q. 

Suppose that vr € Ilu{Mp2niO^))- It is known, at least under the assumptions of Theorem 13.71 
that Q{2ni;p, q; 2n2)(vr) is a unitarizable Harish-Chandra module of Mp2n2i^)- Theorem I4.5l then 
implies that /"(vr) is also a unitarizable Harish-Chandra module in this case. Later, in Ch. 4.4, 
we will show directly that /"(vr) is a unitarizable Harish-Chandra module of Mp2n2i^) in a much 
more general setting. 
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4.3 Sp2ni{^) X Sp2n20^) actioii on Lagrangian Grassmannian 

We are interested in the Mp2n2 x Mp2ni (M)-action on 

r = IndS"^+^"^-"^^^x" 

(see Notations [THl fTT]) . Recall that representation I" consists of sections of a certain homogeneous 
line bundle over the Lagrangian Grassmannian Sp2ni+2n2^) / Pni+n2fl (see Notation [T7|) . 

Notation 19 Z^enoie 5p2ni+2n2(I^)/-Pni+n.2,o ^Z/ 

X parametrizes Lagrangian subspaces of a symplectic space of dimension 2ni + 2n2. We are thus 
led to the problem of analyzing the Sp2ni(^) x 5*^2712 (I^)-a'Ction on X. 

The existence of an open dense Sp2nx^) x 5'p2n2 (I^)-orbit in X can be proved using algebraic 
group theory. But this is not sufficient for our purpose. Not only do we need to know the existence 
of an open dense orbit, but also the detailed structure of this orbit. For this reason, we give two 
elementary lemmas and one theorem in this section. We shall use them in the next section to 
identify the restriction of /° to Mp2ni 0^) x Mp2n2 0^) ■ The main result in this section is Theorem 
K7[ 

Suppose from now on n2 > ni > 0. The case ni = n2 has been studied in [HeO] . |He99] . 
Let me begin by recalling the following result. 

Theorem 4.6 (Theorem 0.3 |HeO| ) Let L(4n) be the Lagrangian Grassmannian of (W^^jQ). 
Then Sp2n(M) x Sp2ni^) CLcts on L{An) with 2n + 1 orbits. Furthermore, there is an open dense 
orbit in L(4n) which CGTi he identified with Sp2n{^^' 



4.3.1 Notations 

Notation 20 Let (M2"2+2ni^ ^-j ^ symplectic space. Let X be the Lagrangian Grassmannian of 
(M2"2+2ni^Q-)_ Fix a standard real basis 

, 62, . . . , 6^2+711 ) /l) f2j ■ ■ ■ /n2+ni } 

on M2rM+2n2 g^^f^ ^/^^^ 

n{ei,ej) = 0, n{fi,fj) = 

where 5^ is the Kronecker symbol. Write 

]R2ni = span{ei, e„,, /i, ...,/„ J 

]^2n2 _ gpan{enx+l^ • • • 1 enj+„2) • • • J /ni+n2}- 

Let Sp2ni(^) be the subgroup o/ 5p2ni+2n2 (^) fixing ej,fj for every j > ui. Let Sp2n2(^) be the 
subgroup of Sp2ni+2n20^) fixing ej,fj for every j < ni. The group Sp2niO^) x Sp2n2(^) is then 
diagonally embedded into Sp2nx+2n2(^) ■ Let 

Vi = span{ei - e„2+i, 62 - e„2+2, • • • e„i - en2+m} 
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Span{fn2+1 + fl, /n2+2 + /2, ■ ■ ■ , /ni+n2 + fni} 

Vo = span{eni+i, . . • .SnJ 

Vq = span{fni+l, . . . fni}- 



Put V = Vi®V2® Vo. 

4.3.2 5*^2712 (IK)-action the Generic Orbit Xq 

The space y is a Lagrangian subspacc of (]K2ni+2n2^ $7), or F G X. Consider the action of Sp2n2{^) 
on y. Let Xq be the S'p2n2 (^) -orbit of ^- Then is a subset of X. 

Lemma 4.3.1 Let Pn2-ni,ni be the maximal parabolic subgroup in Sp2n2{^) stabilizing Vq C M^"'^ 
Let Sp2nii'^)GL{n2 — ni) be the Levi subgroup stabilizing both Vq and Vq. Let N be the nilradical 
of Pn2-ni,ni- Then {Sp2n2(^))v = GL{n2 - ni)N and 



Furthermore, Xq is open in X. 

Proof: Let g G Sp2n2(^)- Then g fixes Cj, fi for every i <ni. Suppose gV = V. 

1. Since g stabihzes V and M^"^^ g stabihzes Vq = V D R^'*^ So 5 G Pn2-m,n 

2. Let j < ni. Since gej = Bj and £?M2"2 =]^2n2^ 

It follows that 



^0 = Sp2n2{^)/GL{n2 - ni)N. 



[{Rej®'R'^''^)nV]. 



Notice that 



(Rej © R^""^) nV = {Rej © R^""^) n (Fi © 1^2 © Vb) = M(e 



'■j ^n2+j) ® 




-j ^n2+j) ® 



3. Similarly, since gfj = fj for j <ni, 



9fn2+j S /n2+j + ^0 



4. Thus, for every 



G span{e„2+i 




5(11) G f + Vq. This shows that g G GL{n2 — ni)N. 
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5. Conversely, if 5 E GL{n2 — ni)N , then g G P„2_„j^„^ C Sp2n2^)- So g fixes ej, for every 
j < ni and 5 stabilizes Vq. Since g € GL{n2 — ni)N, g preserves e^j+j + Vq and /n2+j + ^0 
for every j < ni. So g' preserves — e„2+j + Vq, /n2+i + fj + ^0 for every j < ni. It follows 
that gV = V. 

Therefore, {Sp2n2(^))v = GL{n2 — ni)N and Xq = Sp2n2i^) /GL(n2 — ni)N. To show that Xq is 
open in X, we compute the dimensions of Xq and X: 

dimXo =dim(5p2ni(M)) + dim(Sp2n2 W/5p2ni(M)GL(n2 " ^1)^") 
= dim(5p2ni(M)) + ^ dim(Sp2„2 W/'5P2n,(K)GL(n2 - ni)) 

2 , 2n| + 712 - 2nf - m - (n2 - ni)^ 
=2ni + ni H 

4nf + 2ni + 2?i2 + n2 — 2nf — rii — nf + 2nin2 — n| 
~ 2 

_ nf + n| + 2nin2 + (ni + 712) (4.2) 
2 

_2(ni + n2)^ + (ni + 712) - (ni + 71,2)^ 
~ 2 

=^dim(5p2ni+2n2W/GL(ni +n2,M)) 

= dim(S'p2ni+2n2 0^) / Pni+n2,o) 

= dim X. 

□. 

For 112 = Til, Pn2-ni,ni = Sp2n20^)- Ouv Lemma says that Xq can be identified with Sp2n2(M)- 
This case is already treated in [HeO] (see Theorem 14. 6p . 

4.3.3 5j92ni(K) X 5j92n2(K)- Action 

Next, consider the Sp2ni(^) x 5'p2n2(I^) action on X. Again, our symplectic space is M^^i ©M^"2_ 
The basis for R^^i is 

{ei, 62, . . . Cni, /i, /2, . . . , 

Let 

?7 = Spar2,{e,22+1) 6n2+2) • • • ) Cn2+ni ) /n2+l ) fn2+2i ■ ■ ■ i /n2+ni }• 

Identify M^"! with [/ by mapping 

Cj ^ e„2+j, /i ^ /n2+j) (i € [1, ni]). 

For each g e S'p2ni(I^)! define i{g) G S'p2n2(I^) such that 

K9)\vo(BVi = identity; i{g)\u = g. 

In other words, 5p2ni(IR) is embedded into Sp2n2(^) as the first factor of the Levi subgroup 
Sp2ni(^)GL{n2 — ni). Recall for every g G Sp2ni(^): T{g) is defined to be 

/ni \ f In, 
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Lemma 4.3.2 Let g E Sp2nii^)- Let i{g) he an element in Sp2n2(^) such that i{g)\vo®VQ — 1- 
Regarding {g,T{i{g))) as an element in Sp2niO^) x Sp2n20^)} '^^ have 

{g,T{i{g)))V = V. 

Proof: Let g G Sp2ni{^)- We borrow some ideas from [HeO] . Define an isomorphism : IR^"i U 

by 

H^j) = -en2+j (i = l,2,...,ni) 
4>{fj) = fn2+j (j = 1,2,... ,ni). 
Then Vi ® V2 is the graph of cp and T{i{g)) = (pgcf)^^. Notice that 

{g,T{i{gmVi (BV2) ={{gu,T{i{gmu)) \ u G M^m} 

={{u,T{i{gmg-^u))\ue^^''^} 
={{u,^u) I u G M^"^} 
=Vi ® V2. 



(4.3) 



Thus Vi ® V2 is preserved by the actions of {g,T{i{g))). Clearly, {g,T{i{g))) also preserves vectors 
in Vq. It follows that {g,T{i{g)))V = V. D 

Combining with Lemma 14.3.11 we obtain 

Theorem 4.7 Let Pn2~ni,ni be the maximal parabolic subgroup of Sp2n20^) which preserves Vq. 
Let Sp2niO^)GL{n2 — ni)N be the Langlands decomposition of Pn2-ni,ni cis in Lemma \4-3.1\ Then 
the isotropy group 

{Sp2nM) X Sp2n2m)v = {{r{g),gh) \ g G 5p2ni(M),/i G GL{n2 - ni)N}. 

Moreover, 

Xo = {Sp2nA^) X Sp2n2m)/{Sp2n,m X Sp2n2{^))v 

and Xq is open and dense in X . 

Here Sp2ni(M) and Sp2n20^) are of the standard matrix form. The Sp2ni(M) in the Levi factor 
Sp2ni(M)GL{n2 — ni) of Sp2n2(M) is identified with the standard Sp2niO^)- We avoid using i again. 

Proof: We only need to prove that Xq is dense in X. This follows from an argument similar 
to the argument in [HeO] . Briefly, for every Lagrangian subspace W, define 

Lnd{W) = (dim(T^nM2"i),dim(W'nIR2"2))^ 

Notice that dim(l/ n M^"^) = and dim(y n M^"^) = dim(Vb) = 77-2 — ^1. In general, for every 
W € X, we have 

dim{W n M^"! ) > 0, dim(iy n M^"^ ) > ns - ni . 

We say a Lagrangian subspace W is generic if Ind{W) = (0,n2 — ni). Observe that the group 
action Sp2nii^) x Sp2n2i^) preserves Lnd. Therefore, every element in Xq is generic. 
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Conversely, every generic Lagrangian subspace W must be an element in Xq. Let W be generic. 
Since W n M^^i = {0}, the projection 

restricted to is injective. So dim(p2(^^)) = 't-i + n2- Similarly, since dim(W^nM^"2) = 77-2 — ni = 
dim(PF) — dim(M^"i), the projection 



restricted to W must be surjective. It can then be shown that W is uniquely determined by a 
symplectic map from P2{W) to M^"i, namely piP2^ ■ The kernel of this symplectic map is exactly 
W nM^"2, which is an isotropic subspace. So PiP2^ is reduced to a symplectic isomorphism from 
P2{W)/W n M^"2 to M^"i. By basic linear group theory, the group Sp2ni(^) x Sp2n20^) acts on 
the set of symplectic maps from P2{W) to M?"^^ transitively. So the orbit Xq consists of all generic 
Lagrangian subspaces. Therefore Xq is dense in X. □ 



4.4 /"(tt) and Ind!!?"^^^^ tt^ 

Following Notation \20\ Lemma 14.3. H and Theorem 14. 7^ we set 
Notation 21 

77-2 > ni, P„j+„2^o = Sp2ni+2n2{^)v ■ 

Fix Sp2nii^) X Sp2n20^) C 5'p2ni+2n2 (I^) i'"' Notation \M Write 

Sp2n2{^)v = GL{n2-ni)N. 
Fix a maximal compact subgroup MU{ni + 722) in Mp2ni+2n2i^) such that 
MC/(ni + ns) n Mp2ni W, M?7(ni + 712) n MpanaW 
are maximal compact subgroups for Mp2ni (IR) CLn-d Mp2n2 {^) respectively. Let 

MU{ni) = MU{ni + 712) n Mpa^W, MU{n2) = MU{ni + 712) n Mpana 
Let vr G n(Mp2ni (IR))- Recah from Definition [25] that if 



{(j)i<S>ui,(t)2'S)U2)i = / {T^{gi)ui,U2){I°'{gi)<t)iA2)dgi 

JMp2ni(R) 

converges for every ui,U2 € y(7r), 02 £ then /"(vr) is defined to be (8) ^(vr) modulo 

the radical of ( , )i. /"(vr) is only a (sp2„2(IR), MC/(772))-module. In addition, ( , )i descends into 
a nondegenerate Sp2n2 (^)"™^ariant Hermitian form on I^{tt). The main purpose of this section is 
to show that (/"(vr), ( , )i) is equivalent to the unitarily induced representation 
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Here ( , ) denotes the inner product in the compact picture. We do this in three steps. 



The first step is to show that 



-^"lMp2ni(M)xMp2n2 W ~ ^ (Ai'p2n2 (^^) >< A/GL(n2 -ni ) I det | 2 (g)x",X( 



0) 



where Xq is equipped with a MU{n2) x S'p2ni (IK)-invariant measure. On the right hand side of 
the equation, Mp2n2(^) ^-cts on the left and Mp2ni(I^) acts on the right with an involution r. See 
Theorem 14.81 for the precise statement. The main ingredient of the proof is Theorem 14.71 

From Lemma 14.3. 1^ we have a principal fibration 

Mp2ni(M)/{l,e} ^ Xo ^ Mp2n,{^)/Mp2nM)MGL{n2 - ni)N. 
Let C^{Xqj I^) be the set of compactly supported smooth sections of 

Mp2„2 W ^MGL{n2-ni)N I det | ® ^ ^0- 

The next step is to define a map 



This map is essentially integration on the fiber Mp2ni (IK) (see Definition [29j) . The image of are 
sections of 

Mp2n2{'^) ^ Mp2„{R)xMGL{n2-n^)N 71"^ <^ I det | ^ x"' 

in a proper sense. 

Last step is to extend to y(/") (g) V{tt) under the assumption that ( , )i is well-defined. We 
will further show that ( , )i on /"(vr) coincides with the Hilbert structure on 



^^-t^7i2—ni,ni 



,Mp2 

The main result is Theorem 14.111 Let us first analyze the restriction of I" to Mp2n2 
4.4.1 1° restricted to Mp2ni(K) x Mp2„2(^) 

The representation is a parabolically induced representation. Parabolic induction is a powerful 
tool in the study of representations of reductive Lie groups (see for example (KNj and [Wallach] ) . 
There are the compact picture and the noncompact picture. To study we recall the compact 
picture of 

Notation 22 Let P^j+ni.o be the Siegel parabolic subgroup of Sp2ni+2n2{^) (See Notation [2l\) . 
Define a character on P„2+ni,o = GL{n2 + ni)N by 

_ ni+n2 + l 

exjp{-po{gn)) = \ detg\ 2 [g £ GL{ni +n2),n G N). 

Lift eyi.Y>{— pq) trivially to a character of MPn^+mfi- By abusing notation, denote this character by 
exp(-po)- 
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The Hilbert space of I" consists of sections of the hnear bundle 

Mp2m+2n2(^) XMGL(ni+n2)N C[x'' <^ exp(po)] ^ X. 

Equivalently, 

Definition 26 The Hilbert space of I" consists of functions on Mp2ni+2n2i^) such that 
f{gh) = xihy eM-Po{h))f{g) ( V /i G MP„,+„,,o), 

P dk < oo. 



IMU{ni+n2) 

The inner product between /i , /2 £ I" is 



(/l,/2)= / fl{k)f2{k)dk. 

J MU{ni+n2) 

Notice that the restriction map 

/ G J J\MU{ni+n2) -lnCl^O(ni+n2)^ 

defines an isomorphism between /° and IndjJ^^Q|"^^^^|x"- call the restrictions of f to MU(ni + 
712) the compact picture. 

Restrict the L^-sections to the open dense subset Xq. Fix the base point V (see Notation [20]) . 
From Theorem 14.71 Xq is a single Sp2n2 (IR)-orbit of the form 

Sp2n2m/GL{n2-ni)N. 
We may model ^"U/paniWxMpanaW sections of 

Mp2n2(^) >^MGL(n2~m)N exp(po) O ^ ^0- 

Since Xq is open and dense in X, a section / is uniquely determined by the section f\xo up to a 
set of measure zero. 

Notation 23 Fix the base point V (see Notation \20\) on Xq. Regard f (z I"' as a section on 

Mp2n2(^) ^MGL(n2-ni)N exp(/3o) <8) ^ ^0- 

Let be the function on Mp2n2 corresponding to this section. 

The function satisfies 

fKah) = eM-Po{h))fHg) (V /i G MGL{n2 - n^)N). (4.4) 

Let us consider the action of Mp2n-i(^) x Mp2n2(^) on I". Keep in mind that Mp2ni(K) H 
Mp2n2{^) = {l,e} and 

/"(e, e) = Identity. 
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Lemma 4.4.1 For f G r, gi G Mp2ni(K), 52 S MpsnaC^), (51,52) embedded in Mp2ni+2n2(IK) 
diagonally, we have 

f{{gi,g2)) = f{g2r{gi)-'). 

For every hi G Mp2ni(IK); we have 

r (/il)/]»(52) = /»(52T(/il)). 

Proof: By Theorem 14.71 

/((5i,52)) = /((l,52r(5i)-^)(5i,r(5i)) (4.5) 

= /«(52r(5i)-')x((5i,r(5i))^°exp(-po(5i,r(5i))) (4.6) 
/«(52r(5i)-i). (4.7) 

As to the action of hi G Mp2niO^), 

[/"(/ii)/]«(52) = (r(/ii)/)((l,52)) = f{ihi\g2)) = fKg2r{hi)). 

□. 

So for every hi G Mp2ni(IK)) [-^"(^1)/]" = R{'T{hi))f'^ ■ Here i? stands for the right regular ac- 
tion. Clearly, the action of Mp2n2 (^) on is the left regular action. Now we shall determine the 
Hilbert structure on fK More precisely, for /i,/2 G we want to express (/i,/2) in terms of 

fi A 

Jl) i2- 

Notation 24 Let dk2 he the normalized invariant measure for the maximal compact subgroup 
MU{n2). Let d[k2] be the induced measure on 

MU{n2)/M{0{n2 - ni)U{ni)) ^ Mp2nM)/MPn2-n,,n,. 

Fix an invariant measure dgi on Mp2ni(IK)- Every element x G Mp2n2^) /MGL{n2 — ni)N will 
he represented by k2gi for some k2 G MU{n2) and some gi G Mp2nx{^)- This is the case since 
Sp2n2(^) has a U{n2)Sp2ni(^)GL{n2 — ni)N decomposition. Let dgid[k2\ be the induced measure 
on Mp2n2{^)/MGL{n2 - ni)N. 

Definition 27 Let 

L (Mp2n2(K) -XMGL{n2-m)N | det | (S> x" , dgid[k2]) 
be the set of sections of 

Mp2n2(^) ^MGL{n2-n,)N I det |^ X° ^ Mp2n2 (K)/MGL(n2 - ni)N 

such that 

\\ff= [ |/(A;25i)l'«d5i <oo. 

"'Mp2n2 {R)/MGL{n2-ni)N 

Define 

(/i,/2)»=/ fi{k2gi)f2{k2gi)d[k2]dgi. 

J Mp2n^{«)/MGL{n2-ni)N 

This integral does not depend on the choices of k2. Let g2 G Mp2n2(l^) o,ct on f by the left regular 
action L{g2) and gi G Mp2ni act on f by the right regular action R{T{gi)). 
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Theorem 4.8 Up to a proper choice of dgi, the map f ^ f'^ defines an isometry between I'^ and 



In addition, this isometry induces an equivalence of Mp2niO^) x Mp2n2 0^)-i"^pi"^sentations: 
-^"UfpaniWAfpanaW - ^ (Mp2n2W XMGL{n2-ni)N I det (S> , dgid[k2]) . 

We call this the Mp2n2 (IK)-model of 1°. 

Proof: Let /i,/2 G Clearly, (/i,/2) can be expressed as an integral on Xq by ignoring the 
measure zero set X — Xq. Then (/i, /2) can be expressed as 



Here A{[k2],gi) comes from a product of the Jacobian of a coordinate transform 

Mp2n2{^)/MGL{n2 - ni)N 3 i[k2],gi) [k] G M[/(ni + n2)/M0{ni + na) ^ X 

and an exp{-po) term. Since (/i,/2) = (/"(A:25i)/i, ^°(^25'i)/2) for A:2 G MU{n2) and G 
Mp2ni(IR), ^([^2]>5i)) must be left MU{n2) invariant and right Mp2ni(IR)-invariant. Hence A([A;2], 51)) 
is a constant. The first assertion is proved. 

The second assertion follows from Lemma 14.4.11 □ 



Mp2n2^) ■ We say that a vector f is compactly supported on Xq, if f^ is compactly supported 
on Xq = Mp2n2i^)/MGL{n2 — ni)N. Let C^{Xq,I"') be the set of smooth vectors with compact 
support in Xq. Let C^{Xq, I") j^.f 1/(^^2) MU{n2)-finite vectors in C^{Xq, 1°'). 

Definition 29 Let it G n„(Mp2ni(M)) and u G V{tt). For each f G C^iXQ,!"'), define C^{f(^u) 
to he the function 






Since is compactly supported, for each g2 G Mp2n2(^)^ 



{I%gi)ff{92) vr(<7i)n dg^ = [ f\g2r{gi)) ^{gi)u dg^ 




So C^[f u) is well-defined as a vector valued function on Mp2n2(^)- 
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Theorem 4.9 Let f G C~(Xo, I"). Let vr G n„(Mi?2ni W) e F(7r). Then (g) -u) is an 

element in the Hilbert space oflnd^^p^^^^^^ tt"^ (8)x". Furthermore, 

C\CT{Xo, nMUi2) ® ViTT)) = VilndZ';:^'^^l„y ® X") 
and preserves the (sp2n2 (^)' -^^('^s)) o-ction. 

Proof: Let / G C~(Xo,/°) and n G V{Tr). 

1. Notice that for g2 G Mp2n2(l^) ^^id /ii in the Mp2ni(R) factor of MPn^^ni,ni, 



CP{f®u){g2hi)= / 


(r(<7i)/)«(52/ii) 7r(5i)n d^i 

J) 


-1 


fKg2hiT{gi)) Tr{gi)u dgi 

I) 




fHg2higi) Tr{T{gi))u dgi 

I) 




/*(525i) T^iT{hi^gi))u dgi 

I) 


-j 


f{g29i) vr(r(/ii)"^)7r(r(c/i))« dg 

I) 



Ar{hi)-'){ [ (r(T(5i))/)«(52) 7r{T{gi))u dg^} 

JMp2n^m 



-<r{h,)-')[ 



A^P2ni ( 



{I"{gi)f)Kg2) 7r(5i)« dgi] 



So ® u){g2hi) = 7r(r(/ii)-i)/:0(/ ^ u){g2). 
2. Recall that for every h G MGL{n2 — ni)N C MP„2_„j^„j, we have 

fK92h) = /«(52)xW-"exp(-po(/i)). 

It follows that 



(4.8) 



{I''{gi)ff{g2h)i^{gi)udgi=x{h)-''eM-Poih)) / (r(5i)/)«(52) 7r(5i)n d^i- 

So £0(/ u){g2h) = x(/i)-" exp(-po(/i))>C°(/ u){g2). 
3. Since /'^ is smooth and compactly supported, the restriction of 



g2 



j 



Mp2ni ( 



(/°(5i)/)»(52) 7r(5i)n d^i 



onto the maximal compact subgroup MU{n2) must be continuous and bounded. We see now 
that C^{f n) is a continuous section of 

Mp2n2{^) XMp2„,Q&)MGLin2-ni)N exp(po) <8) x" ^ Mp2„2 W/^;>2ni (M)GL(n2 - Wl)iV. 

Notice that exp(po) = | det | 2 on the GL factor is exactly the normalizing factor. There- 
fore jC°(/ (8> u) is an element in the Hilbert space of Ind^p^"^_^|^^ ^ tt"^ (g) x°- 



54 



4. Since the right action of Mp2niO^) commutes with the left action of Mp2n2{^), preserves 
the action of M'P2n'S^- particular, £° preserves the action of MU{n2). Therefore 

£0 : C^{Xo,nMUin,)<^V{7:) ^ V iIndZ';:^-^*l^7r- ® x"") ■ 

5. Suppose that the map above is not surjective. There must exist a nonzero M?7(n2)-finite 
vector 

<P:Mp2n,{m^n^- 

such that 

•/-te) = (vr" xn{P2^)H92) (V P2 G MP„un^), 
i4>,C'if®u))=0 (V f€C^iXo,nMU(n,),u€Vin)). 
By an easy computation, 

O = (0,£O(/®u))= / {4>{k29i),u)7{h9^d[k2]dgi. 

J Mp2n2{^)/MGL{n2-ni)N 

We see that the M[/(n2)-finite function ^251 {(p{k29i),u) is perpendicular to all compactly 
supported Mf/(n2)-finite functions on Mp2n20^)/MGL(n2 — ni)N with respect to the mea- 
sure d[k2]d9i. So {(p{k29i),u) = for all k2 € MU{n2),9i G Mp2ni G V{n^). Therefore 
^ = 0. We reach a contradiction. Hence 

□ 

4.4.3 The Inner Product ( , )i 

We have shown that £° defines a map from C^{Xq,I") i8) V{n) to Ind^^p^'^^^^ ^ vr"^ x"- In this 

section, we shall extend the definition of £° to (g) V{Tr) under the assumption that ( , )i is 

well-defined for V{P) (g) V{Tr). Let us begin with the following lemma. 

Lemma 4.4.2 Let MU{ni+n2) be a maximal compact subgroup of Mp2ni+2n20^) ■ -^ei lMU{ni+n2) 
be the spherical vector in . Then 

1- 9 iI^i9)'^MU{ni+n2)AMU(ni+n2)) is pOSitivc; 

2. Each (f) G V{I°'), as a function on Mp2ni+2n2(^)> bounded by a multiple of lMU{ni+n2)j 

3. For every (f),ip E V{I°'), there exists a constant C<^,y, such that 

l(^"(5)<A,'0)| < C^^^{I^{9)lMU(ni+n2)^'^MU{ni+n2)) 

for every g G Mp2ni+2n2W; 

4- If {^MU{ni+n2) ^ '^i'^MU(ni+n2) ^ convcrgcs absolutely for all u,v G V{tt), then { , )i is 
well-defined for (g) V{Tr). 
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Proof: In the compact picture, lj\/(/(„j_|_„2) the constant function 1 on MU (ni+n2). {g)^MU (ni+n2) 
is also a positive function. Therefore {I^{g)'^MU{ni+n2)->'^MU(nx+n2)) > 0- continuity, every 
MU {rii + ?7-2)-finite function in the compact picture of is bounded by a positive multiple of 
^MU(ni+n2)- Hence every MU{ni + n2)-finite matrix coefficient of is bounded by a multiple of 

{I (fl')lA/;7(ni+n2)) lM(7{ni+n2))' i-^-, 

for some constants C„ and Cy. As a result, if (^i<8'lMC/(ni+n2)) ^'^lM(7(ni+n2))i converges absolutely 
for all u,v £ V{tt) and lMU{ni+n2) then ( , )i is well-defined for V{I") (S> V{tt). □ 

Lemma 4.4.3 Let (j) G V{I°'). Then there exists a positive constant Ccf, such that \4>^{k2gi)\ < 
(^<l>AlU(n,+n,)i9i) for all k2 G MU{n2) and gi £ Mpam W- 

Proof: First of all 1m[/(„j+„2)(^25'i) = exp(-po(p(5i)) where gi = k{gi)p{gi) under the de- 
composition MU{ni + n2)MPn,+n2,o of Mp2m+2n2{^)- Here gi G Mp2ni(M) ^ Mp2n2W ^ 
Mp2ni+2n2(I^) (sce Notation [2T]) . Similarly, </>''(A:25'i) = 0(^2fc(5'i)) exp(— /9o(p(5'i)). By Lemma 
14.4.21 (2). (j){k2k{gi)) is bounded by a constant for all k2 G MU{n2),k{gi) G MU{ni + n2). So 

for all /!:2 G MU{n2) and 51 € Mp2ni(IR)- □• 

Lemma 4.4.4 Let vr G ntt(Mp2ni(IK)- ^/^e following are equivalent. 
1. L^{tt) is well-defined. 

^- {'^MU{ni+n2) lMC/(ni+n2) ® Converges absolutely for all u,v £ V{tt). 

3. For every u,v £ V^n), the following integral 

/ ^ (^(^(^l))lLc/(ni+n2)(^^l)'lw(ni+n2)(5l))^^P2niW(^(^l)^'^)rf^l 

converges absolutely. Here ( , )Mp2ni(R) ^'^ inner product for L^{Mp2ni{^)) ■ 

4. For every u,v £ V{'7r), the function 

is in Li(Mp2ni(K) x Mp2nA^),dgidhi). 

5. /"(tt) is well-defined for every a. 

Proof: By Definition [27] and Theorem 14.81 we have 

(L(/ll)lMC/(ni+n2), lMC/(ni+n2)) = (-^(^(^1 )) lLc/(ni+n2) ' lM!7(ni+n2)))''- 

The right hand side is equal to 

C(-R(^(^l))lM!7(ni+n2)(5'l)'lMC/(ni+n2)(5'l))Mp2ni(IR) 
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where C is a constant related to the measure of Mp2n2(l^)/-^-fn2-ni,ni- We obtain that 

f i i (4 9) 

=C (^(^ (^1 ) ) lMt/(„i +n2) ) ' ^MU{n, +n2) (^1 ) )mp2„i (M) i^{hl)u,v) dhi . 

So (2) (3). (3) ^ (4) follows from Fubini's theorem for nonnegative functions. 

Suppose /'^(vr) is well-defined. Then {lMU{ni+n2) u,lMU{ni+n2) '■^ ^)i converges absolutely for 
ah u,v e V{7r). So (1) ^ (2). By Lemma (4) , /"(vr) is well-defined for every a. So (2) ^ (5). 
(5) ^ (1) is obvious. We have proved (1) ^ (2) (5) ^ (1). □ 

Suppose that /'^(tt) is well-defined. Let </>, -0 € By Lemma 14.4.31 and Lemma 14.4.41 (4), 



\(p\k2gi)(l)Kk2hi)iTT{T{gi))u,TT{T{hi))u)\dhidgi < oo 

Mp2ni(IR)xMp2ni(IR) 

for all k2 G MU{n2). 

Lemma 4.4.5 Lei Ti be a separable Hilbert space and ^ be a Ti-valued continuous function on 

Mp2ni(M). // 

/ / \{<^{x),^{y))\dx dy < Qo, 

J Jx,yeMp2ni(^) 

then J <I>(x) dx TC is well-defined, as the unique limit of <I>(x) dx as K ^ M]52ni(IR)- Further- 



more, 



(/ ^{x)dx, ^{y)dy)= {<!>{x),^{y))dxdy 

JxeMp2„^{R) Jj/eAfp2ni (K) Jx,y(^Mp2n^{R) 

The proof of this will be given in the Appendix. See Theorem 17.11 

Definition 30 Suppose that I^{tt) is well-defined for a tt £ n„(Mp2ni (I^))- Let (f) G V{I"^) and 
u G V^(7r). Consider the -valued function 

gi G Mp2ni(K) ^ (^Hk2gi)TTiT{gi))u {k2 G MU{n2)) 

Define 

C^{^®u):MU{n2)3k2^ / 4>Kk2gi)^{T{gi))udgi £ H^. 



By Lemma \4-4-5{ C^{4>(^u) is a TC-jr -valued function on MU{n2)- Now is defined on C^{Xq, I°')(> 
y(7r) 0y(7r). 

Theorem 4.10 Let vr G n„(Mp2ni (IR))- Suppose that ( , )i is well-defined for F(7r) 
Then for every u,v V{tt) and V{I"), we have 

2. 2((/>(g)n,^(g)u)i = (/:°((/>(g)n),/:°(V'«)w))ind7r-®x"- 
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In particular, the Hermitian form ( , )i is positive semidefinite and C^{V{I") (S> ^(tt)) = -^"(tt). 

Proof: To prove (1), we will construct a sequence (pm £ C^{Xq, /") such that C^{(t)m ® u){k) 
is a sequence approaching C^{4> ® iJ')ik) in Tin uniformly for all k E MU{n2)- We start with a 
sequence of functions £ C'^{Mp2nx(^)) such that 

1. Hm{kigk[) = Hm{g)y ki,k[ G MU{ni); 

2. UrniexpH) = 1 for all \\H\\ < m where G a and \\H\\^ = \iH,H)\ (see 1.1). 

3. < Hm{g) < 1 for ah g. 

Define (f)m G C^{Xq, 1°^) such that (f>m{k2gi) = 4^^ {k2gi) k'-migi) ■ By Lemma 14.4.31 and the proof of 
Lemma 14.4.51 for the function gi l\j^^^^^^_^.^{gi)T:{T{gi))u, we have 

\\C\(t>m®u){k2)- C\(t}®u){k2)f 



[0L(A:25i)-0"(fc25i)][0fn(fc2/ti)-</'«(^2/Ji)](^(T(5i))^x,vr(r(/ii))n)d5i d/ii 

Mp2ni(K)xMp2ni(K) 



(4.10) 



<4C^ / / |( 
^0 



where ||5i|| = \\H\\ for (71 = kiexp Hk[. So 

£0(0™ (g) u)ik2) ^ /:°(</) ® n)(A:2) 
uniformly for all k2. By the proof of Theorem 14.91 each C^{(f)m ^ u) is continuous in 

Therefore C^{(j) CS) u) is also continuous in IndjJ^p ''^^ vr'^ (8) x"- 



To prove (2), we compute 
2{(j) (g)u,ip ^ v)i 



--2 / (7r(5i)n,7;)((r(5i)</.)(x),V(x))d<7i 

^9lGMp2ni{K) 



(7r(5'i)n,u) / 4)^{k2hiT{gi))il)'^{k2hi)d[k2]dhi dgi 

Si GMp2ni (K) ^ Mp2„2 {K)/MGL(n2 -ni )Ar 

= / {7T{T{gi))u,v) [ f 4>Kk2higiWKk2hi)dh^d[k2\dgi 

(4.11) 

By Lemma 14.4.21 (1), the triple integral ( 14. lip converges absolutely for (p,ip = lAf[/{ni+n2) ^ I^. 
By Lemma 14.4.21 (2)(3). the triple integral ( 14. lip converges absolutely for all 0, "0 ^ V{I°^). So by 
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Fubini's Theorem, changing the order of integrations in Integral ( I4.1ip . we obtain 

2{(j) 0u,^{J iSi v)i 

= [ [[ (l)^ik2higi)^J{i^){n{T{gi))u,v)dhidgi d[k2i 

J[fc2]eAfp2n2{K)/A'/P„2_„-^ J JMp2ni(lR)xMp2ni(IR) 

= / // 4>\k2gi)i;Hk2hi){Tr{T{gi))u,7r{T{hi))v)dhidgi d[h 



[fc2]eMp2n2{K)/MP„2_„-^ J JMp2ni(lR)xM'p2ni( 

{C^{4> ® u){k2),C^{^ ® V){k2)) d[k2] 



[fc2]GMp2n2 {K)/MP„2_„-^ 
= {C^{4> ®u),C^{'ip® V))lnd7r-®x"- 

(4.12) 

It follows that ® V{tt)) ^ /"(vr). □ 

Corollary 4.1 Let vr G n„(Mp2ni (K))- T'/ien for every u,v £ ^(vr) and 0, V' S C~(Xo, /"), 

2((/)(g)ti,V'(8)v)i = (g) ® v))Ind7r^(g)x'»■ 

5'^(ppose that /°('/r) is well-defined. Then for every 4>,iIj £ C^(Xo,/") ® 

By Theorem KM. /"(vr) can be identified with £^{¥{1°") ® V{-k)), which is a subspace of 

T- jMp2n2(R) T ^ a 

Ind;w-p ^ ' TT^ ®x ■ 
It is also iT-finite with respect to the maximal compact subgroup MU{n2). 

AAA A Density Theorem 

In this section, we will show that 



Theorem 4.11 Let r and vr"^ be as in Definition O Let vr € n„(Mp2ni (I^))- Suppose that L^{it) 
is well-defined. Suppose ui < n2 and 7r(e)x"(e) = 1- Let P„2-ni,ni = Sp2ni(M)GL{n2 — ni,M)iV be 
as in Lemma \4.3.1\ Realize MPn2-ni,ni o-s the quotient group 

Mp2„i(M) X MGL{n2 - ni)iV/{(l, 1), (e, e)}. 

Then 

r (^) - c\v{i^) ® = y(indt;^^-(;^^^_^^vr^ ® x")- 

Furthermore, IndjJ^p vr"^ ® is equivalent to the completion of (vr) with respect to the 

inner product ( , )i. 
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By Theorems |M]and EOl /"(vr) ^ £0(^(1°) ® y(7r)) is a subspace of y(Ind^p^"^2W^^^r ^ 

and £*^(C^(Xo, /") (8) Viir)) is a dense subspace of Indj^p^"^_^^^ ^ vr^ (g) x"- Since every element in 
C^(Xo,/") ^(vr) can be approximated by vectors in V{I") ^ V{tt), it is reasonable to believe 
that £^{¥{1") V{tt)) is dense in Ind^fp^^^^^ ^ vr"^ (8> x"- This may be a direct consequence of some 
theorem in functional analysis about the averaging operator C^. Unfortunately, we are not aware 
of any such theorem. We are forced to go back and dig into the compact picture in which the 
Harish- Chandra module is visible. 

Proof of Theorem 14.111 It suffices to prove that C^{V{I°') V{tt)) is dense in 

Ind/w-p ' TT^ «)X • 

We shall prove that every element in C^{C^{Xq, 7°) (g) V{tt)) can be approximated by a sequence 
of elements in £0(^(1") ® ^(vr)). 

Consider ip ® u ^ C^{Xq, 1°') (g) ^(vr). Regard ■0 as a smooth function on MU{ni + n2). By 
the Stone- Weierstrass Theorem, there exists a sequence of functions (pi £ V{I°'), such that (pi ^ iJj 
under the sup norm. In other words, for i sufficiently large. 

By Theorem 14.101 and Corollary 14.11 

(>C°((</>i - ip) 0u),C°{{(j)i - tp) (8)n))lnd7r-0x" = -1p)0U, {(pi - ip) (g)ti)l 

The latter is no greater that 26'^{lMU{ni+n2)'^'^^ ^MU{ni+n2)®'^)i with respect to I^(SnT. Therefore, 
£^{(pi (g>u) ^ C^{ip (g) u) under the Hilbert norm. By Theorem SJl C^{C^{XqJ'') ® is 
already dense in IndJJ;^^^"^^^^^^^ vr^ (g x". So C^{V{I'') ® V{ti)) is dense in IndJIJ^'"^ W x 

By an easy TT-finiteness argument with respect to MU{n2), 
We have proved that 

r(vr)-Indl5^p^-(^) vr-^X^ 



□ 

Combining this with Theorem 14.51 we obtain 

Theorem 4.12 Assume that a = ni + n2 + 1 (mod 2). If n e Uu{Mp2ni{^}) and /°(vr) is 
well-defined, then 



p+q=ni+n2+l, p—q=a (mod 4) 



as unitary representations. 
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4.5 Wave Front Set and the Nonvanishing Theorem 

Theorem 4.13 ( |BV85| jVOOl] ) Let P = MAN be a parabolic subgroup of a reductive Lie 
group G. Let a be an irreducible representation of L = MA. Then 

WF{lnd$a) = cl{lndfWF{a)). 

Notice that lnd^{WF{a)) may contain several irreducible components of the same dimension. Ap- 
plying Theorem 14.131 to I" for Mp2n(M), we obtain 

Corollary 4.2 The wave front set WF{I'^) = Indj2^'JJ^O[in] . 

Again assume p + q = n + 1. By the theorem of Kudla-Rallis-Lee-Zhu, we have 

/"= e{p,q;2n){trwial). 

p+q=n+l,p—q=a (mod 4) 

Consider the wave front sets. From Corollary 12.31 lndg|'^^^|^^0[xn] consists of n-|- 1 components D^-^^ 
and is equal to the closure of Op"] nsp2n(I^)- We have 

c1(IJl>(^))= IJ WF{9{p,q;2n){tiwial)). 

p+q=n+l,p—q=a (mod 4) 

This implies that these WF{9{p,q; 2n)(trivial)) must be in the closure of O^nj nsp2n(I^) ( |BV80| ). 
In fact, they are distinct. 

Theorem 4.14 Assume p + q = n + l. Let [2"]^*^ be the signed Young diagram with i rows starting 
with +. Then 

T4^F(6l(p,g;2n) (trivial)) = ^(C'pnjM U O^njcp-i)). 
Here we take Oj2'i](-i) and C)j2n](n+i) to be the empty set. In particular, 

M/F(/") = IJ T^F(e(p,g;2n) (trivial)) 

p+q=n+l,p—q=a (mod 4) 

Proof: From Lemma 13.5.31 only cZ(Oj2„](p) U 0|2"](p -i)) can occur in the image of the moment map 
m2 for {0{p,q), Sp2nO^))- Thus by Theorem l3.11l of Przebinda, 

W F {9 {p,q;2n) (trivial)) C c/(Oj2„](p) U O^njcp-i)). 

But by the decomposition Theorem 14.41 

n n 

U d(0[^2]) = WFin = \J WF{e{p,q;2n){trivial)) C \J d{0^^„^u)) 

j=0 p+q=n+l,p~q=a (mod 4) j=0 

We must have W F {9 {p,q;2n) (trivial)) = cZ(Oj2"](p) U C'[2"](p-i))- ^ 

This theorem partly explains the decomposition theorem of Kudla-Rallis-Lee-Zhu in terms of the 
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orbit philosophy. Generahzing this idea to /"(tt), we give a proof of the Generic Nonvanishing 
Theorem. 

Proof of Theorem 14.21 By Lemma 14.4.21 (3), every MU{ni + n2)-finite matrix coefficient of /" is 
bounded by a multiple of 

MU{ni+n2)i lM(7(ni+n2))) 

which is a spherical function. Notice that the infinitesimal character of is equal to 

n-i+n2 



_.ni+n2 — l ni+ 77,2-1 ni + n2 - 3 ni + n2 - 3 ni + n2 - 1 ni+n2-l.. 



and 

711+712 



. ^^ — no —Ti-i — no — ni — no ,ni +no, 

- p{Mp2n,+2n, (M)) < ( \ \ \ + 1, • • • , \ + [^^], 0, • • • , 0) . 

By Theorem 11.11 or by Harish-Chandra's spherical expansion, (/''(5')lM!7(ni+n2)' -'-M!7(ni+n2)) is 
bounded by a multiple of 

In particular, (/°(5i)lM;7(ni+n2)> lMC/(»ii+n2)) is bounded by a multiple of 

Since vr € i?cjs(-/\^P27ii (I^); '^('^i + "-2 + 1,0; 2ni)) (Def. [TS]) . for every leading exponent w of vr, 

^ ni - 1 - p{Sp2m 



We obtain 

- (^. ^ - 1. ... , ^ + 1) + 2p(Mp,„,(M)) 
^n^-n^ - (R)) - (=1±^, ^ - 1, ... , ^ + 1) + 2p(Afe„. (M)) ^^^^^ 

n2-ni-l ni+n2 ni+n2 ^ "-2 - ni 
^ ^ ( 2 ' 2 ' ■ ■ ■ ' 2 + 1) + "1 - 1> • • • 1) 

-<0. 

Inequality 14.131 and Theorem 11.11 imply that ( , )i is well-defined for V{I^ ® V{Tr)). So /"(vr) is 
well-defined for any a. Suppose that a = ni + n2 + 1 (mod 2). Theorem 14.121 applies. We have 



I^^S; ® = Q(2ni ; p, g; 2n2 ) (vr) 

By Theorem 



'■MP 

p+g=7ii+7i2+l,p— (?=« (mod 4) 



CpT™^^^x" = es{p,q;2n2)esi2n,;p,q){7T). 

p+q=ni+n2+l,p—q=a (mod 4) 



62 



It follows that 

I^<!';(S((n.-nO^^(^^) = U WF{es{p,q;2n2)es{2n,,p,q){7:)). 

p+q=ni+n2+l,p—q=a (mod 4) 

Let On be a nilpotent orbit of maximal dimension in 

Define S C {(p, q) | p + g = ni + n2 + l,p fixed parity} to be the set of those (p, q) such that Od 
occurs in the image of the moment map m2 associated with (0(p, q), Sp2n2 0^)) (see Definition [25]) . 
By Theorem 13.111 On can only appear in WF{9s{p,q;2ni)9s{2ni;p,q){Tr)) with (p, g) G S. There 
must exist at least a {po, qo) G S such that 

On C VFF(0,(po,go;2n2)es(2ni;po,'?o)(vr)). 
Hence Q(2ni;po, 9o; 2n2)(7r) / and 6's(2ni;po, go)(7r) / 0. □ 

Under certain favorable circumstances, On only occurs in the image of m2 for a unique {p, q) 
with p + g = ni +n2 + 1. In this case, we are able to determine for which (p, q), 9s{2ni;p, q){Tr) ^ 0. 



Corollary 4.3 Consider the group Mp2ni+2n2{^) with ni < 712. Let n be a unitary representation 
in 

Rss{Mp2n, (M),a;(ni + n2 + 1, 0; 2ni)), 
(see Def. \T8\) . Let On he a nilpotent orbit of maximal dimension in 

Fix a parity of p. Suppose that On is not contained the image of the moment map m2 associated 
with (0(p,ni + 722 + 1 - p), 5p2„2(K)) for p / po- Then Q(2ni;po,ni + n2 + 1 - po;2n2)(7r) / 
and 9s{2ni;pQ,ni + n2 + 1 -po)(7r) / 0. 
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Chapter 5 



Construction of Unipotent 
Representations, Unitarity and 
Infinitesimal Character 

Start with a real nilpotent orbit O = Od in U{0{p,q)) or U{Mp2n{^)) (see Definition [T2]) . First, 
construct an alternating sequence of nilpotent orbits of metaplectic groups and nilpotent orbits of 
orthogonal groups: 

0{di) = OT,,0{di - 1) = Od-1, . . . , 0{1) = Oo-di+i. 

Write 

= G{0{1)), G{2) = G{0{2)), G{d,) = G{0{d,)). 
Clearly, 0{k) G U for every k (see Cor. [22]). Let C be the negative of O 

O' = {-x\x£ O}. 

O' may or may not equal to O. 

Definition 31 We define M{Oy)) inductively. 

1. For 0{1), let J\f{0{l)) he the set of all one- dimensional unitary characters of G{\). 

2. Suppose N{0{k)) is defined. 

(a) For G{k + I) orthogonal, define N{){0{k + \)) to he 

{■K{k+l)®r} \ 7r(/c + l) = 9s{G{k),G{k+l)){TT),-K G M{0{k)),ri a character of 0{p,q)}. 

Let M{0{k + 1)) = MQ{0{k + 1)) U {vr* I vr G Mo{0{k + 1)')}. 
(h) For G{k + 1) metaplectic, define No{0{k + 1)) to he 

{7r(fc + 1) I ^(A: + 1) = ds{G{k),G{k + l))(7r), vr G M{0{k))}. 

Let Mi{0{k + 1)) = A/'o(0(A; + 1)) U {vr^ | vr G AAo(t(0(A; + 1)))} and 

M{0{k + 1)) = Mi{0{k + 1)) U K I vr G Mi{0{k + 1)')}. 
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Any of the operations in the definition of A^, tensoring with a character, involution by r or *, do 
not alter the infinitesimal character, the real part of the leading exponent, the unitarity, or the 
associated variety of the annihilator. In most cases, we will ignore these operations. The reader 
should be aware of the fact that * and r operations alter the wave front set by the involution / and 
T. The author would like to thank the referee for suggesting this definition. 

Theorem 5.1 For Or, € U{G), the set J\f{0-£)) is not empty. 

For technical reasons, we will postpone the proof till the next chapter. In this chapter, we work 
under the assumption that AA(C'd) 7^ 0. We will show that 

Theorem 5.2 LetOD be inU{0{p,q)) orU{Sp2n{^))- Then anyTr{k) € M{0{k)) is in Rss{G{k),uj) 
(Def. \18\) for the pair (G{k),G{k + 1)). The set A/'(Od) is well-defined. The representations in 
AA(C'd) are all unitary. 

Thus Af{Or>) C IIu{G{Od)). We will further determine the infinitesimal character of vr G A/'(C'd)- 
It turns out that X(vr) only depends on d and consists of p-like segments. 



Let Od be in U{0{p,q)) or h({Sp2nO^))- The group G(Od) can be read off from D^, the number 
of positive boxes in D, and from D~ , the number of negative boxes in D. Construct an alternating 
sequence of symplectic signed Young diagrams and orthogonal signed Young diagrams: 



D(di) = D, D(di - 1) = D(di) - 1, ... , D(2) = D(3) - 1, D(l) = D(2) - 1. 
Let D(0) = 0. Recall that ||d|| stands for the number of boxes in d. We have the following. 
Lemma 5.1.1 Let Od G ^- Then 
1. For every k € [l,di — 1], 

D{k + 1)+ - D{k)+ > D{k)- - D{k - 1)-, D{k + 1)- - D{k)- > D{k)+ - D{k - 1) 



5.1 Unitarity of 7^(0) 



Thus, 



D{k + l) 



+ 



+ D{k-iy > \\d{k)\\ 



D{k + l) 



+ D{k-1)+ > \\d{k) 



2. 



For every /c € [1, di 



D{k + l)+ > D{k) 



D{k + 1) 



> D{k)+. 



3. 



For every k € [l,di — 1], 



d(A: + l)|| + ||d(A: 



1)11 > 2||d(A;)|| +2 



if'D{k) is symplectic; 
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4- for every k E [1, di — 1], 

||d(A; + 1)11 + ||d(A:- 1)11 > 2||d(fc)|| 

ifT){k) is orthogonal. 
5. ||d(A: + 2)|| = ||d(/c)|[ {mod 2) for all k > . 

Proof: All the statements can be read off from D. The number of + boxes on the k + 1-th column 
must be greater than or equal to the number of — boxes on the A:-th column. We obtain (1). Adding 
all columns together, we obtain (2). (3) and (4) are consequences of (2.a)and (3. a) from Definition 
[T2l (5) follows from (2.c) and (3.c) from Definition [T2j □ 

Proof of Theorem I5.2t Let Oj) G U. Consider the sequences D(A:) and 0{k). We shall prove 
that Tr{k) is unitary and lies in Rss{G{k),uj)) for the pair {G{k),G{k + 1)). We will use induction. 

1. For A; = 1, by definition, every 7r(l) in J\f{0{l)) is unitary. If G{1) is metaplectic, then we 
have 

. D{i)+ = D{ir- 

• -0(2)+ > 2Ll(l)-,i:>(2)- > 2D{1)+, by Lemma EXU (1); 

• D{2)+ + D{2)- > 40(1)- + 2, by Lemma [5XT] (3). 

if G(l) is orthogonal, then we have D{2)+ = D{2)' > D{1)- + D{1)+. It follows from the 
Definition [TSl that every unitary representation is in Rss{G{l),uj) with respect to the dual 
pair (G(1),G(2)). 

2. Suppose that Af{0{k - 1)) is well-defined and M{0{k - 1)) C Uu{G{k - 1)). Suppose that 
M{0{k - 1)) C RssiGik - 1), w) with respect to the pair {G{k - 1), G{k)). This is our induc- 
tion hypothesis. 

If G{k) is metaplectic, let 

p' = D{k + l)+,g' = D{k + iy,n = D{k)+ = D{k)-,p = D{k - l)+,g = D{k - 1)". 

Then by Lemma 15.1.11 p + q = p' + q' (mod 2), p' + q' — 2n > 2n—p — q + 2, p' > n and q' > n. 
Clearly max{p' , q') > n. lin ^ min(p', g'), then Theorem 13.61 holds and ■n{k) = 6s{'ir{k — 1)) is 
well-defined and unitary. Furthermore, ^{k) G Rss{G{k),tLi) with respect to {G{k),G{k + 1)). 
If n = mm{p' , q'), without loss of generality, assume, n = p' . We shall prove that this case 
does not occur if Od £ ^• 

Since D{k)~ = n = p' = D{k + l)"*", the diagram D(A; + 1) must have its first column 
marked with — . By the structure of signed Young diagram, D(/c) must have its first column 
marked with +. D(A;) must have its second column marked with — . Furthermore, the first 
column of D(A;) and the second column of D(A;) must have the same length. Otherwise, there 
are at least two rows of length one in D(fc). At least one H would occur in the first column 
of D(A;) according to our definition of yD-{n,n). This contradicts the fact that the first 
column of D(A;) is all marked with +. Thus, 

||d(fc)||-||d(fc- 1)11 = ||d(A;- 1)11 -||d(A;-2)||. 

By the same argument, D(A;) must be of the following shape and sign pattern: 
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+l- 


l+l- 


l+l- 


+l- 


l+l- 


l+l- 


+l- 


l+l- 




+l- 


l+l- 




+l- 






+l- 







In other words, the j-th column of D(/c) must be marked by (—I)-' and the 2j + 1-th column 
of D(fc) must be of the same length as the 2j + 2-th column. This kind of D(A;) is excluded 
from U. So the case n = min(p',g') does not occur. 



If G{k) is orthogonal, let 

n' = D{k + 1)+ = D{k + 1)- , p = D{k)+,q = D{k)~ 
Then by Lemma I5.1.H we have 

2n' — p — q>p + q — 2n>p + q — 2n — 2, 



n = D{k - 1)+ = D{k - 1)' 



p > n, 



> n. 



By the same argument as for metaplectic G{k), n ^ min(p, g). By Theorem 13. 7^ ■n{k) = 
6s{Tr(k—l)) is well-defined, unitary and 7r(A;) G Rss(G{k),uj) with respect to the pair {G[k),G{k+ 

1))- 

3. It follows that M{Oik)) C w) for the pair {G{k),G{k + 1)) and every Ti{k) G 

N{0{k)) is unitary. 



□ 



5.2 Infinitesimal Character 



Theorem 5.3 (Thm 1.19, |PR96j ) Let 2{7r) be the infinitesimal character of n. Suppose that 
IT occurs in the theta correspondence with respect to {Sp2n0^),O{p,q)). 

1. Suppose p + q < 2n. Then I{9{p, q;2n){TT)) can be obtained by augmenting I (tt) by 

p + q p + q rP + 1^ P + l^ 

[n — ,n 1, . . . , 1 + 



2 2 ' ' ' 2 ' 2 ' 

2. Suppose 2n + \ < p + q. Then 2{9(2n;p, g)(vr)) can be obtained by augmenting T{tt) by 

(?^_„_l.!l±i_„_2,...,l^-[^l) 

3. Suppose p + q = 2n or p + q = 2n + 1. Then X{9{p, q; 2n)(7r)) is just X(7r). 
Notation 25 We define the orthogonal segment 



I^{m) = i--l,--2,. ..,--[-]) 
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and the symplectic segment 

[^] 

, " . 

^ , , ,m m „ m , . /n + 1 ^ , 

The orthogonal segment 1^{m) is just p(o(m, C)). For m even, T_(m) is /9(5p^(C)). For m odd, 
2r_(m) is the infinitesimal character of the oscillator representation of Mpm+i(^)- 

Theorem 5.4 Let On be in U{0{p, q)) or U{Sp2nO^))- Let tt G M{Ot>) and 

d* = (mi > m2 > . . . > Tn(ij). 

// G(D) is an orthogonal group, then 

X(7r) = (X+(mi),X_(m2),X+(m3),X_(TO4), • • •)• 
// G(D) is a symplectic group, then 

I{7r) = (X_(mi),2:+(m2),X_(m3),X+(m4), . . .)• 

Proof: Let vr E A/'(Od)- We prove this theorem by induction on di. If di = 1, by definition of 
M{0{1)), TT restricted to the identity component of G(Od) must contain a trivial constituent. Thus 
I{tt) = X±(mi). Suppose our assertion holds for any D with di < k. Let di = k + 1. 

If G(D) is 0(p,q), then according to our definition of tt, '^\sOo(j>,q) ™ust be equivalent to 

e{G{Oj,^,),G{Oj,)){a)\sooM 
or its contragredient for some a G A/'(Od-i)- 2^(7r) can be obtained by augmenting I{cr) with 



mi , "zi mi mi 

(^-1,^-2,...,— -[—])=J+(mi). 

If G(D) is Mp2n(K), observe that I{7r^) = T(7r). Then 

for some cr G A/'(Od-i)- ^(tt) can be obtained by augmenting 1(a) with 

[^] 

mi mi mi mi - 1 

□ 
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Notation 26 Let 

d* = {nil ^ "^2 > • • • > fT^di)- 
For each orthogonal Young diagram d, we define X-|_(d) to he 

(J+(mi),2:_(m2), J+(m3),T_(m4), . . .)• 

For each symplectic Young diagram d, we define I-{d) to be 

(T_(mi),T+(m2),X_(m3),2'+(m4), . . .)• 

Theorem 15.41 states that I{it) = I{d) for n G M{Or>). Thus I{'it) only depends on d, not on the 
signs of D. 

5.3 ^±(d) and d: An algorithm 

In this section, we will seek a direct way of obtaining I±{d) from the Young diagram d. We assume 
d* is very odd or very even. The Young diagram d does not have to be pre-rigid or in U. The 
algorithm can be described as follows. 

1. First, cover the Young diagram d by horizontal and vertical dominoes 

m □ 
□ 

as follows. We cover each column of the Young diagram d by consecutive vertical dominoes, 
starting from the bottom row. If d* is very even, vertical dominoes covers d completely. If 
d* is very odd, we have the first row left uncovered. We cover the first row of the Young 
diagram d by consecutive horizontal dominoes, starting from the right. We may have the 
leftmost block uncovered. In that case, cover it with a horizontal domino anyway. We call 
this domino an open domino. 

2. For each vertical domino DO, we can enumerate the number of dominoes above it. A hori- 
zontal domino will be counted as ^ domino and a vertical domino will be counted as a full 
domino. Thus we obtain a number n{DO). 

3. If d* is odd, fill the open domino with no number and fill the other horizontal dominoes with 
^. For X_|_(d), if a vertical domino DO is in the k-ih. column, we fill in DO with the number 

n{DO) + For ^-(d), if a domino DO is in the A;-th column, we fill in DO with the 

number n{DO) + ^ ^^^^ ■ Extracting the numbers in all the dominoes, we obtain 2{d). 

Notation 27 Let A = (Ai, A2, . . . , A„) G M". We define A to be the reordering of A such that 

Ai > A2 > . . . > An. 
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5.4 Orderings and Reversal Phenomena 



Recall from }CMj that complex nilpotent orbits of classical groups have a partial ordering ^. 
Notation 28 Write di ^ d2 if O^j is contained in the closure of O^^ . 

The ordering < for Young diagrams is different from the ordering ^ for numerical sequences (see 
Notation [2|). If one regards and d\ as numerical sequences arranged in descending orders, then 
di ^ d2 if and only if d^ ^ d^. 

Theorem 5.5 (Reversal Phenomena) Let Odi cLnd O-q^ nilpotent orbits of a fixed orthog- 
onal group or symplectic group. Suppose that di* and d2* are either both very even or both very 



odd. If di ^ d2 then I±{di) h I±{d2) (see Notation \27\ ). 



Later we will show that V{Ann tt) = Od for every vr G M(D). This theorem suggests that, among 
the representations A/" of a fixed group G, the smaller the infinitesimal character, the bigger is the 
associated variety and conversely. For example, the trivial representation has the smallest associ- 
ated variety, namely {0}. Its infinitesimal character is given by p{G), the greatest among all I{tt) 
with TT G M. Theorem 15.51 will be used in the Appendix to establish the estimates of T±(d) given 
in Theorem 16.11 and Theorem 



We prove the reversal phenomena through a number of lemmas. The conditions in the theorem 
are always assumed in these lemmas. 

Lemma 5.4.1 // the Young diagram di can be obtained from the Young diagram d2 by moving 
one domino to its lower left without changing other dominoes, then di ^ d2. 

We call this procedure a move. 

Lemma 5.4.2 If the Young diagram di can be obtained from d2 by a finite number of moves, then 
di ^ d2 and vice versa. 

This lemma holds only under the assumption of the theorem. If d* is very even and dg is very odd, 
this lemma is no longer valid. 

Lemma 5.4.3 // Young diagram di can be obtained from the Young diagram d2 by a move, then 
2±(dO >= 2±(d^. 

This Lemma is an easy consequence of our algorithm from d to I±{d). 



By the above three lemmas. Theorem 15.51 is proved. □ 

Corollary 5.1 Let <ind Od^ be pre-rigid nilpotent orbits of a fixed orthogonal group or sym- 
plectic group (see Definition Suppose that d\ and dg are both very even or both very odd. 
Then I-|-(di) ^ Z-|-(d2) if and only if di ^ d2. 

Proof: For pre-rigid orbits, one can reconstruct d* from I{d) in a unique way and the partial 
orderings are reversed in this reconstruction. □ 
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Chapter 6 



Associated Varieties and Existence of 



Let Cd G ^{G) and vr G A/'(Od)- In this chapter, we study the matrix coefficients and the 
associated variety of vr. We first bound T{tt) by a multiple of p{G). By Theorem 11.21 we obtain a 
bound on the matrix coefficients of vr. Then we apply Przebinda's theorem to show that 



With V{Ann vr) in hand, we gain some control of WF[tt). Our next task is to prove Theorem 15.11 
Let me briefly describe the proof of Theorem 15.11 Suppose vr(fc) € J\f{p{k)). Then T{'ir{i)) and 
V{Ann vr(i)) are all known for every i < k based on our computation. If G{0{k)) is orthogonal, then 
Af{0{k + 1)) 7^ follows from Theorem [331 If G{0{k)) is the metaplectic group, Af{0{k + 1)) 
follows from Theorem 14.21 and Cor. 14. 3[ The details of the proof of 15. H are given at the end of this 
chapter. 




V{Ann vr) = cl{Od). 



6.1 Estimates on Infinitesimal Characters: I 



Let G = Mp2„,(M) and p{G) 



(n,n - 1,. . . ,1). Let Or, be in U{G). Let 
d* = {mi > m2 > • . . > rrid^ ) 



Then we have 2n = ||d||. 



Theorem 6.1 Suppose Od G l({Mp2n{^))- Let 



(mi,m2, . 



Then 



T_(d) ^ 



mi + 2 



p(Mp2n(M)). 



2n 



A similar statement holds for G = 0{p,q). 



Theorem 6.2 Suppose Od eU{0{p,q)). Let 



d = (mi, m2, . . . 
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— — mi + 2 p + q p + q p + q + 



In fact, we will have that 



mi 



T_(d) ^ ^p(Mp2n(M)) 



p + q — 2 2 2 2 2 



(6.1) 



for Oj) € U. These two statements are slightly stronger than those of Theorems 16.11 and 16.21 In 
the Appendix, we will give a proof for the first statement and Theorem 16.11 We skip the proof for 
Theorem 



6.2 Associated Varieties of J\f{0-[)) 

Theorem 6.3 (Estimates on Leading Exponents) Let Od be inU. Let d* = (mi, m2, . . . , m^^). 
Let vr G A/'(Od). 

1. If G{0-d) is Mp2ni^), then every leading exponent v of tt satisfies 

"fiiv) -< (^^^ - l)p(Mp2„(M)). 

2. If G{0-£)) is 0{p,q), then every leading exponent v of it satisfies 

fi{v)^{ 1'^\ -l)piO{p,q)). 
p + q-2 

Proof: Part (1) is a direct consequence of Theorem 16.11 and Theorem 11.11 Part (2) is a direct 
consequence of Theorem 16.21 and Theorem 11.11 □ 

Theorem 16.31 can be established independently in the framework of |Heq| without resorting to 
the infinitesimal character estimate in Theorems 16.11 and 16.21 However, the argument in |Heq| 
relies heavily on the estimates of twisted integral. It is more general but less intuitive. The estimate 
of the the leading exponents using infinitesimal character only works for the unitarily small rep- 
resentation, namely unitary representations with infinitesimal character sitting inside the convex 
hull of {wp I w G W{q, f)))}. This method can be used to establish strongly semistable condition 
for unitarily small representations. 

Theorem 6.4 Let O-q G U{G). Let vr G M{0-d). Then V{Ann vr) = d(Cd)- 

Proof: Notice that none of the operations in the definition of A/'(Od) changes ^(vr) or V{Ann tt). 
Without loss of generality, we assume 7r(/c) = 6s{G{k — l),G{k)){Ti{k — 1)) and 7r(di) = vr. Recall 
Theorem 13.91 and Theorem 13.101 

1. Condition (1) is automatic by the definition of {G(k — l),G{k)). 

2. Conditions (2) from Theorem 13.91 and 13.101 are satisfied due to Theorem 16.31 and the definition 

of {G{k-l),G{k)). 
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3. 7T{k — 1) is not a discrete series representation. If 7r(A; — 1) is in the discrete series and G{k — 1) 
is of rank r, then the infinitesimal character I{Tr{k — 1)) ^ (r — 1, r — 2, . . . , 1, 0) (for example, 
see Theorem 9.20 |KNj ) . By Theorem 16.11 and Inequalities ( I6.ip this is only possible for 
several large T(7r(/c — l))'s. For these large X{'n{k — l))'s, the representation 7r(/c — 1) can not 
be in the discrete series. 

4. Condition (4) is assumed. 

5. Condition (5), which basically say that 9s{T^{k — 1)) is unitary, is readily verified by Theorem 

6. Conditions (6) are checked in Lemma 13.6.11 and Lemma 13.6.21 

Thus Theorem 13.91 and Theorem 13.101 hold. Therefore, by Lemma 13.5.21 of DKP, 

V{Ann TT{k)) = Q{G{k - 1), G{k)){V{Ann TT{k - 1))) = cl{0{k)). 
By induction we have V{Ann vr) = cl{Od)- D 

Corollary 6.1 Fix an Ov, G U{Mp2n{^))- Let O{po,qo) = G{Or>-i). Let vr € A/'o(C'd)- Then 
there exists a nilpotent orbit in WF{7r) such that 

1. do = d. 

2. G{0-D,-i)=0{po,qo). 

Proof: Suppose that vr = 6s{po,qQ;2n){a) with a G AA(Od-i)- Then V{Ann a) = Od-i and 
V{Ann vr) = O^- By Theorem 13.111 WF(7r) must be in the image of m2 with respect to 
{0{po, go) , Sp2nO^)) ■ Then Lemma 13.5.31 says that every orbit Os in WF{tt) satisfies G(Os-i) = 
0{s,t) with s < po and t < qq. There must be at least one Odq in WF{-ir) such that G(C'do-i) = 
O{po,qo). Otherwise, V{Ann it), which is the complexification of WF(tt), will be strictly smaller 
than cl{0(i) and will not be equal to d(OD)- D 

6.3 Nonvanishing of J\f{0-£)) 

Let On G U. Let tt € A/o(C'd)- We have proved that vr must be unitary and V{Ann vr) = Od- In 
other words, any representation in M{0) must be unitary and V{Ann vr) must be equal to cl{Od)- 
In this section, we will prove that 7V(C'd) is nonempty. In fact, we will prove that A/'o(Od) is 
nonempty. 

Theorem 6.5 Let Od G Z^- Let O{po, qo) = G{k - 1), Mp2n, (M) = Gik) and 0{p, q) = G{k + 1). 
Suppose 

7r{k - 1) G M{0{k - 1)), and 7r{k) = 6*^(^0, Qo; 2ni)(7r(/i; - 1)) / 0. 
Themr{k + l) = es{2ni;p,q){Tr{k)) / 0. 

Proof: Since Or> G l^, write 

d(A; + 1)* = (m-i > m2 > > nn > ...,). 
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Then 

d(fc)* = (m2 > 771-3 > ^4 ^ •••))) ^(^ ~ 1)* = ("^3 > "^4 — ••■!)' 

p + g = 2ni + mi, 2ni = + Q'o + "12- 

By Theorem 16. 4|, V(^nn 7r(A;)) = Od(fe) a^id V(^nn7r(/c — 1)) = Od(fc-i)- Let 712 = p + g — ni — 1. 
Clearly n2 — ni = p + q — 2ni — 1 = mi — 1 > and ni + ^2 + 1 > p + By Theorem 14.11 and 
Theorem 15.21 it suffices to show that 



Qi2ni;p,q;2n2){TT{k))^0. 

1. By Corollary 16. H there exists a nilpotent orbit Odq of Sp2ni(M) such that 

(a) do = d{k); 

(b) Do — 1 contains po [+] 's and qo IZ] 's; 

(C) Odo ^ WF{7T{k)). 

2. By Theorem liT3l 

Observe that 71-2 — ni = ?Tii — 1, and mi — 1 > m2. m2 is the first entry of dp = d{kY. By 
Theorem ESI 

^"%2„^{M)©gl(n2-ni,R)^v(-^Doj 

contains mi — m2 irreducible components 



I ,mi-m2-l/-/i 



And S^-'^ can be obtained by 



• merging 2 columns of length mi — 1 to Dq from left; 

• extending the signs of Dq for the first m2 rows; 

• assigning j I ' l+l 's and mi — m2 — 1 — j l+l ' I 's to rows of length 2. 

3. Notice that for every j, 

(s(j))* = s* = (mi - l,mi - I,m2,m3, ...,); 

(s^-^) - 1)* = (s - 1)* = (mi - 1, ma, ms, . . . , ). 

Consider the signature of S^-'^ — 1. For the first m2 rows, there are po + m2 [+] 's and qo + m2 
[Z] 's. For the last mi — m2 — 1 rows of length 1, there are j E 's and mi — m2 — 1 — j H 's. 
Thus the signature of S'^-'-' — 1 is 

(po + m2 + j, qo + m2 + mi - ni2 - I - j) = (po + m2 + j, qo + rrii - I - j) 
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4. Fix the parity of p. Let j = p — Po — i^2- Here p — Po is equal to the number of [+] in the 
first two columns of D(A; + 1). It is greater or equal to the number of boxes in the second 
column of D(A; + 1). It follows that j = p — Po — m2 > 0. Since p + q = mi + m2 + Po + qo, 
the signature of S^-'^ — 1 equals 

{po + m2+j,qo+mi-l- j) = {p, qo + mi - 1 + po + m2 - p) = {p, q-l). 

Prom Lemma 13.5.31 Og(j) occurs in the image of m2 associated with {0{p, q), Sp2n2 (^))- Also 
from Lemma I3.5.3( C'g{j) does not occurs in the image of m2 associated with 

{0{p -2i,q + 2i), Sp2n2 (K)), {0{p +2i,q- 2i), Sp2n2 W) 

for any i 7^ 0. 

5. By Theorem 15.21 Tr{k) E Rss{Mp2niO^),i^ini + 712 + 1,0; 2ni)). So the generic nonvanishing 
theorem and consequently Cor. 14.31 hold. Therefore, 

Q{2ni;p,q;2n2){7r{k))^0. 

and 

TT{k + l)=esi2ni;p,q){TT{k))^0. 

□ 

Proof of Theorem I5.lt It suffices to prove that Afo{0) is nonempty. We prove this by induc- 
tion. The existence of 7r(l) is trivial. The existence of Afo{0{2)) is established by Jian-Shu Li. 
Suppose that /c > 2 and 7r(A;) € Af{0{k)). If G{k) orthogonal, from Theorem E31 ^f{0{k + 1)) is 
nonempty. If G{k) is metaplectic, from Theorem 16.51 TT{k + 1) is not zero. Thus Afo{0{k + 1)) is 
not empty. □ 

6.4 Perspectives 

In this paper, we only treat 0{p,q) and Mp2nO^)- Quantum induction for other classical group of 
type I can be defined similarly. Excluding the pair (0(m, C), S'p2n(C)), the groups are connected 
and the metaplectic lift on these groups split. Thus the discussion on these groups should be 
considerably easier. We do not intend to work out the details here. Rather, we shall list some 
problems concerning quantum induction and unipotent representations for classical groups of type 
I. We will also point out problems in connection with representation theory and the theory of 
automorphic forms. The main question is whether quantum induction can supplement parabolic 
induction (including complementary series) and cohomological induction ( [K- Vj ) to produce a 
complete classification of n„, Hauto and Urama- In any case, this article should be viewed as a 
starting point for new development in that direction. 

6.4.1 Wave Front Sets of A/'(0) 

Wave front sets under Howe's local theta correspondence are discussed in detail in [PROP] . There 
are still open questions which need to be answered. For the unipotent representations in Af{0), 
we have showed that their associated varieties are the complexification of cl{0). Now one may 
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speculate that cl{0) is the wave from set of vr E M{0). This is far from the case. In fact, there are 
irreducible representations in M{0) such that WF[tt) ^ cl{0) as demonstrated by Theorem 14.141 
What should be true is that O C WF{ti). We fall short of proving it. 

Conjecture 3 For every vr G N{0), O C WF{tt). 

Let me give some hint on how one may proceed. Przebinda started a program attempting to obtain 
the Harish-Chandra character of 6s{tt) from the Harish-Chandra character of vr. If his program 
works in our situation, we will have a confirmation of this conjecture. Let me point out a critical 
step. 

Conjecture 4 Let tt e Rs{MGi,MG2). Then 

y(0,(AfGi,MG2)(vr)) ^ [Homg,,MKiK, 7r)]MK2. 

Roughly speaking, this is saying that every MK2-iinite vr'^-valued MGi-equivariant distribution on 
uj can be obtained by integration over MGi as in [HeOO] . 

Notation 29 Let On he a nilpotent orbit of G. Let Hoj^iG) be the set of n n(G) such that 
cZ(Od) is of maximal dimension in WF{Tr). Let Hu^Oui^) ~ ^Oui^) ^ n„(G). 

According to a conjecture of Barbasch-Vogan ( |BV85j ). 

{tt G Uo^iG) I ||J(7r)|| = min{||T(a)|| | a G Uo^iG)} C n„(G). 

Assume that Conjecture [3] is true. Then 7V(C'd) ^ ^u,Ou{^)- iact, we conjecture that the 
representations in A/'(Od) are the ones that occur in the Barbasch-Vogan conjecture. 

Conjecture 5 Let G be either an orthogonal group or a metaplectic group. Let vr G Ilc)j^{G) with 
Od rigid. Let A be the shortest infinitesimal character among all Z{'k) with vr G lioui^)- Then 
\ = l{d). 

This conjecture is false if fails to be rigid. Now assuming Conjecture El we formulate the 
following conjecture which implies Conjecture [5l 

Conjecture 6 Let O be a rigid orbit inlA. Suppose that G{1) is not 0{l){l > 2). Then M [On) 
exhausts all Yiu,0'D{G) for Mp2n{^) and 0{p, q). 

Notice that the rigid orbits in U include all special rigid orbits. For nonspecial rigid orbits of 
0{p,q), by the works of Brylinski-Kostant ( [BK]) and Huang-Li ( [HL] ) , there might be unitary 
representations of the nontrivial covering of SOq{p, q) attached to them. Our construction does not 
cover their cases. However, for rigid nilpotent orbits in Conjecture (U our construction should be 
exhaustive. 

6.4.2 Wave Front Sets and Induction Functors 

A critical observation in this paper is that D — 2 remains to be a symplectic (orthogonal) Young 
diagram if D is a sympletic (orthogonal) Young diagram. On the philosophical level, this paper 
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raises the question whether there always exists a functor 

This question does not always have a positive answer. For example, there are Od with n„ OoiG) = 
0. 

Problem 1 For all real forms of symplectic and orthogonal groups, construct functors 

whenever Uu^OniG) 7^ 0- Here is a group attached to ©0-2 that is of the same type as G. 
We call each one of these functors, an induction functor. 

The second question is whether one can construct enough number of induction functors J-{0-[)-2, Od) 
such that all vr G Iiu,Ou{G) can be obtained from Ilu^oT:,-2iG^) through one of the induction func- 
tors. 

Indeed, if is induced from Cd-2 5 then unitary parabolic induction functor is the right candidate. 
Under different assumptions, complementary parabolic induction and cohomological induction can 
also be applied ( [BOl] [K- Vj ) . This situation has been studied intensively in the past. 

The case that Od is not induced from 00-2 is the main focus of this paper. It seems that 
the quantum induction Q is the right choice. For example, one can study the following problem. 

Problem 2 Consider Mp2n{^)- Let Od be a symplectic nilpotent orbit. Suppose that On is not 
induced from 00-2- Let 2n2 = ||d||, (p, q) = ((D — 1)+, (D — 1)^) and 2ni = ||d — 2|[. Then is 

Q(2ni;p,g;2n2) : Uu,o^_,{G) ^ Uu,OuiG)'^ 

Notice that Ilu,OuiG) may be empty for some D. Even in those cases, this problem seems to make 
sense, that is, Q(2ni;p, g; 2n2) will conjecturally map Ilu,Ou~2(.G) to zero. We shall come back to 
this problem in a subsequent paper. 

6.4.3 Unitary Dual, Automorphic Dual and Ramanujan Dual 

Let G be a classical group of type I. Let HhermiG) be the set of irreducible Harish-Chandra modules 
with an invariant Hermitian structure. Let Iltemp{G) be the set of irreducible Harish-Chandra 
modules with almost L^ matrix coefficients. Recall that 

n(G) D UhermiG) D H^G) D Utemp{G). 

The classification of the admissible dual n(G) is due to Langlands. Langlands proved that every 
vr G n(G) occurs as the unique quotient of certain induced representation from a tempered repre- 
sentation. The quotients are often known as Langlands quotients. The classification of Utemp is 
due to Knapp-Zuckerman. The classification of the Hermitian dual is also known. See for example 
Theorem 16.6 and its remark [KN) . 
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With these classifications in hand, classification of unitary dual can be translated as the deter- 
mination of unitarity of the Langlands quotients. But this approach is of great mathematical 
complexity. It often involves the positivity of certain analytically defined intertwining operator. In 
|VU86j and |VU87j . Vogan envisioned a more geometric approach based on Kirillov-Kostant's or- 
bit philosophy. We believe that quantum induction should be sufficient to supplement the existing 
techniques to produce a classification of the unitary dual for classical groups of type I. 

Problem 3 Arrange a process to construct irreducible unitary representations of a classical group 
G using unitary parabolic induction, cohomological induction ( \K- V|/ ] and complementary series 
construction, combined with quantum induction. Can this process be arranged to produce all irre- 
ducible unitary representations? 

Recently, Salamanca-Rib a and Vogan conjectured that all irreducible unitary representation can 
be constructed by cohomological method from an irreducible unitary representation of a smaller 
group except unitarily small representations |SVj . Notice that all the representations in N{0) 
are unitarily small and quantum inductions seem to keep the infinitesimal characters small. So 
investigations on Problem 3 may give some hope to a classification of the unitary dual of classical 
groups of type I, modulo the conjecture of Salamanca-Riba and Vogan. For coverings of classical 
groups, the situation is a little more delicate. But the same idea should apply. 

Alternatively, one can follows the route of Problem 1 and construct the unitary dual by apply- 
ing different induction functors based on the wave front sets. 

Theta correspondences as formulated by Howe ( |Ho79| ) originated from the theory of theta se- 
ries ( |We65| . [Si]). The history of theta series went back to the theta functions defined by Jacobi. 
Special cases of theta series were studied by many people in number theory. The upshot is that 
theta correspondences should produce automorphic representations of a bigger group from auto- 
morphic representations of a smaller group. This line of ideas was further explored by Rallis, Li 
in some quite general settings ( |Ra87] . |Li94j ) and by many other people in special cases. We are 
tempted to make the following conjecture. 

Conjecture 7 All representations in J\f{0) are automorphic. 

In fact, we can even say more about N{0) and quantum induction. Recall that 

n„(G') D iiauto[G) D n 

rama 

We adopt the definitions from [BLSJ. In principle, according to the conjectures of Ramanujan and 
Selberg ( [Se], [Saj), complementary series should not occur in Ilauto{G) or Ilrama{G). Assuming 
that. 

Problem 4 Camr S Ilauto{G) be constructed by unitary quantum induction, unitary cohomological 
induction and unitary parabolic induction? 

The argument that parabolic inductions should send Hauto of a Levi subgroup to Ilauto{G) was 
given in [BLSj with their H taking to be the parabolic subgroup rather than the Levi subgroup. 
Finally, for split classical groups, Barbasch showed that the irreducible spherical unitary repre- 
sentations can be constructed as a parabolically induced representation from a complementarily 
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induced representation tensored with a special unipotent representation ( |B01j ). The Ramanujan 
dual is simply the intersection of the automorphic dual with the spherical dual. Thus one may ask 

Problem 5 For G a split classical group of type I, can one construct all vr € ^rama{G) by unitary 
quantum induction and unitary parabolic induction? 

This exhaustion question is perhaps too difficult to answer. For SL(2,M) where no quantum 
induction is involved, this is equivalent to the Ramanujan-Selberg conjecture. 
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Chapter 7 

Appendix 



7.1 Infinitesimal Characters for Mp 

Recall that ^^-^(d) consists of an arithmetic sequence with multiplicities. For d* odd, the sequence 
ends with ^; for d* even, the sequence ends with zero if d 7^ [1]^". In this chapter, we shall give a 
proof for Theorem 16. li We start with a lemma. 

Lemma 7.1.1 (Triviality) 

1. If X ^Ji and X' ^ fi' , then X + X' ^Ji + fi' . 



2. // A ^ 77 and X' -< /x', then (A, A') -< (/i, fx') . 

3- If Xi < for all I, then X ^Ji. 

4- If Xi < Hi for all I, then X ^Ji. 
Let G = Sp2n(^)- Then 



p(G) = (n,n-l,...,l) =T_(2n). 



Lemma 7.1.2 Suppose that m = r (mod 2). If r <m, then 



m 



il(m),l^ir)) < X_(m + r). 

m + r 

Proof: Write 

B = {Z(m),l+{r)), A= T_(m + r). 

m + r 

We prove our lemma by induction on If ^^^^ = 0, m = r. Then 

m m — 1 m — 2 m — 3 2 1 

2' 2^' 

m m — 2 m — 2 m — 4 m — A 
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B ends with (1,0) if m is even and (i, i) if m is odd. Obviously, Aj. > B^. So B ^ A. Assume 



>2' 2 

that m — 2 > r and 

m — 2 



(I_(m - 2),I+(r)) ^ I_(m + r- 2). 

m — 2 + r 



Notice that < We have 



,171 



B = {l_{m)a+{r)) = - 2)a+{r)) 



,m m — 2 _ , 
< (— , X-(m + r - 2)) 

, m m + r m ^ , 

^ ( , I^(m + r - 2)) 

m + r 2 m + r 



(7.1) 



m + r 
□ 

Lemma 7.1.3 Consider a partition of 2n 

2j 



I-{m + r) = A. 



d = [m, m, m — 2, m — 2, . . . , m — 2j + 2, m — 2j + 2, mo,r) 
with m = rriQ = r (mod 2) and m — 2j > mo > r > 0. T/ien 



m 



T_(d) ^ ^X-(2n). 

Proof: Clearly, 2n = mo + r + (2m — 2j + 2)j. If j = 0, d* = (mo,r). We have X_(d) ^ 
^^^ T_(mo + r) by Lemma 17.1.21 Assume j > 1. 

Use induction on m. When m = 2, 5(2,0,0) = 2r_(2) ^ -^-(2). So our lemma holds for m = 2. 
Assume our lemma holds for m — 1. 

Suppose that r > 1. The case r = can be treated similarly. Obviously, 2n < 2m(j + 1). 
So 

m(2n — 2j — 2) = 2mn — m(2j + 2) < 2nm — 2n = 2n(m — 1). 
It follows that ^ < Consider 

, / s 

e = (m — 1, m — 1, m — 3, m — 3, . . . , m — 2j + 1, m — 2j + 1, mo — 1, r — 1). 



Suppose that m is even. Then X-(d) = (X_(e) + ^, 0, 0, . . . , 0). By induction hypothesis. 



m — 1 



^-(^)^2;73^^-(2--2.-2). 
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Then we have 



1 m — 1 1 

J_(e) + - -< I-(2n - 27 - 2) + - 

^ 2 -2n- 2j -2 ^ ^ 2 

m— 1 m — 1 1 
-(0,l,...n-j-2) + 



2 2n-2j-2' ' ' '2 

mm, 

^ 0,1,... ,n- 7 -2 

- 2 2n ^ ' 

mm, 

m 



m / V 

=-^{n,n-l,...,j + 2). 



We obtain I-{d) ^ ^T-{2n 



2n" 

i+ 1 
1 1 1' 



Suppose m is odd. Then X_(d) = (2r_(e) + -,-,...,-). We will again have 

n-j-l 

1 m 



2:-(e) + - ^ — (n,n-l,...,i + 2). 
As I changes from n — j to n, ^2r_(2n)/ — i decreases from positive to negative. 



2ra 



^^X_(2n),-j;Z_(d). 

8=1 i=l 



increases and then decreases. It suffices to show that 



^^X_(2n),-^T_(d),>0 

1=1 i=l 



Notice that 

v^/^ / „ (m — 2s + 1)^ v^/^ / „ (m — 2s 
5^(X_(m - 2s))i = ^ Y.^I+{m - 2s))i = ^ - 
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We obtain 

n n 



2n 

i=l 1=1 



_m {n+ l)n i-^ {m - 2s + if {m - 2s - if {niQ + l)^ (r - l)^ 
~2^ 2 8 ^ 8 ^ 8 8 

1 

= -[m{2n + 2) - ( 2(m - 2s)2 + 2) - (mo + 1)' - (r - l)^] 

i-1 i-i 

=-{m(mo + l + r- l + 2 + V 2(m - 2s)) - (mo + 1)^ - (r - 1)^ - 2j - V 2(m - 2s)^} 
8 — ^ — ^ 

s=0 s=0 

1 

= -{(mo + l)(m — mo — 1) + (r — l)(m — r + 1) + 2(m — j) + 2 ^^(m — 2s)(m — m + 2s)} 
>0 



Therefore T_(d) r< ^l^{2n). By induction, J_(d) ^ ^1^{2n) for all m. □ 

Proof of Theorem 16.11 Fix n first. Let G Z//(Mp2n(I^))- Fix the number of row mi = m. 
Consider 

do = (m,m,m - 2,m - 2, ... ,m - 2j + 2,m - 2j + 2,mo,r) (j > 0) (7.4) 

with 

< r < mo < m — 2j — 2, m = mo = r (mod 2). (7.5) 
Here j = means that dg = (toq, r). By Lemma 17.1.31 

m 



T_(do) r< ^^-(2n). 

Notice that Cdo is the minimal orbit in lA{Mp2n{^)) with a fixed m. Therefore, by Theorem 15. 5^ 
for any d with m rows, 2r_(d) ■< I_(do). We obtain for any Oa G hl{Mp2n(^))^ 

□ 

7.2 Integration of Functions with Values in Hilbert Spaces 

In this section, we shall give a proof of Lemma 14.4.51 By changing coordinates, it suffices to prove 
the following Lemma. 

Lemma 7.2.1 Let Ti be a Hilbert space and ^ be a Ti-valued continuous function on M". // 

[(<I>(x), <l'(y))| dxdy < oo. 
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then J^^{x)dx & Ti. is well-defined as the unique limit of = ^{x)dx for any increasing 
sequence of compact sets Ki — > M". Furthermore, 



(/ <^{x)dx, ^{y)dy)= {^{x),^{y))dxdy 

Proof: Let <I>(x) be a continuous function from M" to 7i such that 

/ |($(x), $(y))| dxdy < oo. 

Jx,y£M" 

Let -ftTj be an increasing sequence of compact sets Ki — > R". Then as i — oo, 



\mx),^{y))\dxdy 
Define = /j^. ^{x)dx. Then for any m > i, we have 



(uKm - UK,,UKm - UK,) =( / ^{x)dx, / ^{x) dx) 



(7.6) 



= / {^{x),^{y))dxdy 

J{Km-K,)xiKm-Ki) 

< [ \mx),^y))\dxdy 

<[ \i<^ix), <^{y))\ dx dy ^0. 

Therefore {uk,} is a Cauchy sequence and hmui^, exists. So u = / ^{x)dx £ H is well-defined. 
Furthermore, \\u — uki\\ 0. By a similar argument, we have 



(n,n) = J {<^>{x),<i>{y))dxdy. 
We can generalize Lemma 17. 2. li as follows. 

Theorem 7.1 Let H be a Hilbert space and X a locally compact space. Let ^ he a cr— finite regular 
Borel measure. Let ^ he a Ti-valued continuous function on X. If 

\{^{x),^{y))\d^l{x) diJL{y) < oo, 

x,y&X 

then J^^{x)dfi{x) (z "H is well-defined as the unique limit of Jj^^{x)dfi{x) for any increasing 
sequence Ki with fi{X — UKi) = 0. Furthermore, 



(/ $(x)(i/x(x), / ^{y)dn{y)= {<^{x),<l>{y)) d^i{x) dfi{y) 

Jx&X Jy&X Jx,y£X 

The convergence of ^{x) d^{x) is neither a strong convergence nor a weak convergence. We refer 
to it as convergence on compact sets. 
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